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About the research 

Apprenticeships and vocations: assessing the impact of 
research on policy and practice 

Jo Hargreaves, National Centre for Vocational Education Research 

This report assesses the impact of NCVER’s research against two themes: the role of 
apprenticeships in a modern economy (focusing on work relating to completions) and the 
nature of the vocations and competencies required by industry. The evaluation spans work 
published from 2005 to 2015 and examines the impact of 32 publications, complemented by 
input from a number of stakeholders who were asked to report on their use of NCVER’s 
research outputs. The evaluation, based on a combination of quantitative metrics and 
qualitative methodology, shows how NCVER’s publications are influencing policy and 
practice in the vocational education and training (VET) sector. 

Key messages 

■ The analysis identified: 

42 553 downloads of the reports from NCVER’s website/ portal 
2952 full record visits on VOCEDplus 

632 citations across various sources, including policy and legislation 
217 references across an assortment of media outlets. 

■ While metrics alone do not directly signify impact, they do suggest NCVER has a sizeable 
knowledge footprint, with the publications in this study gaining widespread attention 
within Australia and internationally. 

■ In terms of policy: 

Evidence was found of the research directly influencing six discrete policy 
documents or legislation and informing numerous submissions across 30 national 
or state/territory reviews. These submissions often led to an appearance by 
stakeholders or NCVER at parliamentary Senate House of Representative 
hearings. 

Stakeholders also reported a research impact on polices designed to address 
apprentice completion and commencement rates, the way in which pre- 
apprenticeship programs are structured, industry training strategies and 
mentoring support for apprentices. 

■ With respect to practice: stakeholders suggested the research has informed better 
recruitment and pastoral care practices for apprentices. The vocations research has 
influenced several professional development programs for VET practitioners. 

While the interplay between research output and policy and practice is never 
straightforward and there are many factors influencing policy decisions, connections do 
occur. This report in particular has highlighted the remarkable variety in which NCVER 
publications are used and valued, based on the views of responding stakeholders. 

Dr Craig Fowler 
Managing Director, NCVER 
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A Introduction 

This report presents the findings of a case study assessing the influence of a select number 
of research and statistical publications on policy and practice in the area of the following 
two important themes: 

■ the role of apprenticeships in a modern economy 

■ the nature of vocations and the competencies required by industry. 

This is done using the framework for measuring research impact developed by NCVER 
several years ago (Stanwick, Hargreaves 6t Beddie 2009). 


NCVER’s vision: to 
inform and influence 
vocational education 
and training in 
Australia through 
credible, reliable and 
responsive research 
and statistical 
services. 


Mixed views exist, and persist, on definitions of impact and the purpose of assessing the 
impact of research. These are coupled with contested opinions on the ability to measure the 
impact of research effectively and efficiently. Nevertheless, NCVER and others continue to 
refine the task. Indeed, this is third occasion on which we have undertaken an impact 
evaluation, each time enhancing our approach. 


Over many years, and since our original study, the impact agenda has received substantial 
attention. Of note is the Higher Education Funding Council for England, which undertook a 
sizeable initiative to assess the quality of research in higher education institutions under the 
Research Excellence Framework . In Australia a national commitment to the exercise 
received less support in the early days. This was followed by various attempts such as that 
undertaken by the Australian Technology Network, along with the Group of Eight 
universities, who led a trial to assess the impact of research produced by the university 
sector. An impact model for universities is now receiving renewed attention as a result of 
the Australian Government’s commitment to support both pure and applied research, 
outlined in the National Innovation and Science Agenda . In March 2016 an Australian-first 
panel of experts was announced. They will lead efforts to measure the value of research, 
working with a specialist group appointed to design the engagement and impact measures 
for the 2018 Excellence for Research in Australia exercise. 


The approaches taken in the United Kingdom and Australian impact studies, as well as the 
continuing conversations among international experts published on the London School of 
Economics and Political Science impact blog, suggest NCVER’s framework and overall 
approach can be considered good practice because of the focus and combination of 
quantitative and qualitative indicators. We use metrics from a wide variety of sources and 
gather informed opinions and personal testimonies from end users. This results in a richer 
narrative about the impact of the research than the numbers alone can tell. 

NCVER’s mission is to inform and influence policy and practice in the Australian vocational 
education and training (VET) sector through credible, reliable and insightful information, 
coupled with balanced analysis and insights, to optimise the benefits arising from publicly 
funded research endeavours. Our philosophy is that for research to have impact it must be 
seen as relevant by its target audience, make a useful contribution to knowledge, be 
impartial, adopt rigorous methodologies and use reliable data, be conducted ethically, be 
above criticism of bias, be interesting and be read or at least become known through 
summary articles, conferences and the like. 
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We are fortunate that the very nature of our work, which requires direct interaction with 
our stakeholders, puts us in a good position to explore and evaluate influence. Indeed, the 
critical role of intermediary and independent organisations to increase research translation 
and collaboration activities between researchers and end users was highlighted in a recent 
study examining, across fourteen countries, ‘what works’ in knowledge-translation policies 
and measures (Australian Council of Learned Academies 2015). 

This does not mean the task is straightforward. Conceptual, practical and methodological 
obstacles complicate the job of assessing the diversity of outcomes from social science 
research. Furthermore, research findings come into use in complex ways (Davies, Nutley & 
Walter 2005). 

Consequently our approach is realistic, recognising both the imperfect role of metrics in 
research assessment and the applied and social science focus of our research efforts. Yet 
our purpose for doing so remains clear. Observing and understanding the nature of our 
influence helps us to produce the evidence, in ways that best meet the needs of 
stakeholders in the training system. 

Impact and engagement across NCVER 

The analysis in this report is confined to a case study across two themes, a total of 32 
publications and a 15-year time period. This represents only a glimpse of the total research 
and statistical output generated by NCVER. 

While we recognise the importance of using a valid framework to assess impact on the scale 
contained in this report, we also regularly collect and monitor a range of other metrics 
within the company aimed at assessing use and engagement across the full breadth of our 
research and statistical outputs. 

One such measure is keeping track of the H-index on all NCVER-authored publications. The 
H-index, named after the developer Hirsch, is a measure of academic impact that has 
generated widespread interest. It is an author-level metric that attempts to measure both 
the productivity and citation impact of the publications of a scholar. The advantage of the 
H-index is that it combines an assessment of both quantity (number of papers) and quality 
(impact, or citations to these papers). Hirsch calculated the H-index of Nobel prize winners 
and found 84% of them to have an H-index of at least 30. Newly elected members in the 
National Academy of Sciences in Physics and Astronomy in 2005 had a median H-index of 
46. 1 

The H-index 

The H-index has been found to have considerable face validity, although there are critics 
and no real agreed rating spectrum. 

As at March 2016 NCVER’s H-index is 47, which can be considered very good to outstanding. 


1 <http://www.harzing.com/publications/white-papers/reflections-on-the-h-index>. 
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^ About the framework 

Background 

NCVER’s approach is based on the Payback Framework, developed in the 1990s as a way of 
assessing the impact of health research (Buxton & Hanney 1996). In keeping with this model 
we define ‘impact’ broadly in terms of its direct, indirect or incremental use and influence. 
It may be conceptual in terms of affecting levels of knowledge or understanding, attitudes 
and beliefs, or instrumental in terms of changing policy or practice. 

It is important within any research impact framework to recognise how knowledge is used. 
Establishing a connection between research findings and policy is particularly difficult 
because of the political context in which policy development occurs. Even where knowledge 
that stems from systematic research and analysis can be shown to contribute to policy 
change, the route may be gradual, subtle and diffuse. Knowledge ‘provides a background of 
empirical generalisations and ideas that creep into policy deliberations’ (Weiss 1980, 
p.381). Where this is the case, policy-makers will most likely be unaware of how concepts or 
insights from research have played a direct role in shifting their thinking; nor is there a 
strong culture of citing research in policy documents. 

We are also interested in the extent to which non-government stakeholders use conceptual 
evidence from the research to challenge or support policy positions. This has implications 
for what should be assessed. In particular, a body of research associated with a specific 
topic potentially carries more influence compared with one research report, although this is 
not always the case. 

With all this in mind our approach is pragmatic and modest. It was developed, and has 
evolved, amidst full recognition of the various challenges associated with how knowledge is 
used and how that use can be measured, especially in the social sciences. This is in 
comparison with the hard sciences, where concrete measures associated with observable 
change may result over time. In health, for example, reductions in disease occurrence may 
form part of a research impact model. 

We recognise impact can be positive or negative, may or may not be recognised, can occur 
over short or long periods of time, and is highly dependent on context. Research that ‘fails’ 
to have impact at one point in time may be considered highly relevant at another point. Not 
every single independent research report will have direct and obvious benefits but an 
absence of indicators or citations does not mean an absence of activity or influence, while 
prolific output does not necessarily mean quality. 

The framework 

NCVER’s framework measures impact against four categories: 

■ producing knowledge: using metrics associated with research outputs to determine the 
extent to which research is being accessed and knowledge dispersed 

■ building capacity: assessing the extent to which the research has improved the abilities 
of researchers and stakeholders to engage with the research 


Only occasionally 
does research supply 
an answer that policy 
actors employ to 
solve a policy 
problem (Weiss 
1980 ). 


NCVER 


11 


■ informing policy : judging if and how research has been useful in informing or guiding 
policy, primarily through a bibliometric search and interviews or surveys with end-users 

■ informing practice : establishing how the research has informed or guided practice, 
again through a bibliometric search and interviews or surveys with end-users. 

There are two components to the framework: a flowchart (or logic model) of the research 
process (figure 1); and a multi-dimensional categorisation of impacts, which offers a useful 
way of thinking about where impact can occur across the various research stages and as a 
guide for collecting evidence (table 1). 

Figure 1 Logic model for assessing impact 



Source: Stanwick, Hargreaves and Beddie (2009). 


The framework places a significant emphasis on embedding an impact strategy into the 
research cycle from the beginning. The conduct of research itself is not an isolated activity: 
it can be a stage when impact occurs; for example, by way of work-in-progress seminars or 
consultations with end-users who have a vested interest in the research findings. The key to 
this strategy is taking end-users on the research journey, without compromising the 
independence of the research effort. 

Importantly, the approach acknowledges a number of key elements on which NCVER places 
very high importance. These include: 

■ establishing the priorities and purpose of the research with involvement from end-users 

■ engaging with stakeholders throughout the research cycle 

■ adopting multiple dissemination strategies when the research findings are known, 
acknowledging the diverse needs of the sector. 

The framework distinguishes between two types of impact indicators: outputs, which 
include publications and the like, and outcomes, less immediate impacts. For each of the 
categories of impact, table 1 gives examples of types of impacts, outputs, outcomes and 
sources of information (evidence). It needs to be noted that the table describes possible 
indicators. This does not mean that they will all apply to all case studies; nor are they an 
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exhaustive list - there may be less generic indicators that are specific to one piece of 
research or to one field of study. 

Sources of information and ways of capturing them are proxies for impact; they are not 
‘measures of impact’. This points to the difficulties in developing a suite of indicators for 
some categories of impact. While measuring citations can be a relatively straightforward 
exercise, identifying more amorphous elements of the research process, such as networks 
and collaborations, is more difficult, making accurate measurement of their impact less 
easy. 


...a mature research 
system needs a 
variable geometry of 
expert judgement, 
quantitative and 
qualitative indicators 
(Wilsdon et al. 2015). 


Validating the quality of research 

While considerable attention is given to quality assurance across all research and statistical 
activities and publication processes at NCVER, one aspect missing from our original impact 
framework was an independent validation of the quality and potential impact of the 
research. 

For this study we have trialled the use of NCVER’s Editorial Board, esteemed Australian and 
international academics in the tertiary education sector, as an external assessment panel 
(see appendix B). 

The approach was inspired to some extent by the way in which expert panels were used in 
the Research Excellence Framework across universities in the United Kingdom. The intention 
of our trial was to determine the extent to which our impact model could be enhanced, 
with the use of an independent judgment of the research, beyond metrics and the self- 
reported use of research by stakeholders in our case studies. The Editorial Board members 
were asked to rate a number of the reports under consideration according to quality, as 
defined by originality and rigour, potential impact in terms of reach and importance, as well 
as general accessibility. 
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Table 1 NCVER’s research impact framework: categorisation of impact indicators 


Categories (domains) 
of impact 

Specific types of impact 

Outputs: immediate indicators 
(push factors) 

Outcomes: medium-long-term 

indicators 

(uptake factors) 

Sources of information (evidence) 

Knowledge production 

Dissemination strategies 

Raising awareness of and 
engagement with the research 
Increasing knowledge and 
contributions to the literature 

Ability of research to inform future 
research 

Publications of various types: 
reports, good practice guides, At a 
Glance publications, research 
overviews, fact sheets, infographics 
etc. 

Presentations to stakeholders 

Media (press releases) and social 
media (tweets, blogs) 

Data sharing 

Citations 

Research used in vocational or higher 
education courses as 
readings/references 

Broad bibliometric and citation analysis of various 
publications and articles using VOCEDplus, Publish or 

Perish, Google, Google Scholar My Citations tool and 
Hansard 

Google analytics (web data on number of downloads by 
stakeholder) 

AltMetrics* (Alternative Metrics) using BuzzNumbers 
(social media, blogs, twitter) if established in advance of 
an impact study 

H-index 

Media coverage (where, requests for interview) 

Numbers of subscribers (Insight, Twitter, Portal) 

Stakeholder self-reported use of research 

Quality validation of the research using NCVER Editorial 
Board assessments 

Capacity-building 

(research-related 

impact) 

Supporting the skills and abilities of 
researchers to undertake fit-for- 
purpose research from both the VET 
and higher education sectors 

Improving the skills of relevant 
stakeholders engaging with research 
and enhancing their decision-making 
abilities 

Mentoring and supporting early career 
researchers 

Formation of partnerships and 
collaborations (academic, TAFE, 
industry etc) 

Presentations to stakeholders and 
peers 

Invited keynotes 

Professional development 
opportunities 

Technical/data specific publications 

Ongoing or new collaborations and 
networks established 

Career development of researchers 
Leadership development 

Subsequent grants received 

Recognised as expert in field 

Improved quality of research 
funding submissions 

Records of collaborations kept 

Evidence of new or follow-on research based on skills and 
previous research history 

Public recognition (awards, invited keynotes) 

Publication of reports by early-career researchers 

Informing policy 

Raising profile of research among 
decision-makers (government, 
industry, other stakeholders) 

Provides information base for decision 
makers to practically, critically and 
confidently debate an issue 

Influence and involvement in decision- 
making processes 

Publications (see above) 
Presentations at policy roundtables, 
workshops and other policy-related 
events 

Submissions to parliamentary or 
other enquiries 

Policy briefing notes 

Research used or quoted in 
government reports or by ministers 

Requests for information 

New or modified policies or plans that 
guide decisions or actions 

Active participation in policy networks 
(think tanks) 

Citations: parliamentary submissions and Hansard 
references and other policy-related documents 
(Government white papers etc., noting research may be 
used but not always cited) 

Logs of information requests, records of presentations and 
meetings, memberships of groups maintained by 
researchers 

Qualitative case studies (direct contact with those who can 
validate use of research): interviews and/or surveys of key 
informants 

Informing practice 

Raising profile of research among 
practitioners (providers, industry, other 
stakeholders) 

New or modified practices broadly 
encompassing behaviour, actions and 
knowledge of how things are done 

Good practice guides 

Fact sheets 

Research overviews 

Practitioner developed publications 

Information requests by practitioners 

Use of good practice guides 

Adoption of new or changed practices 
Networks and collaborations 

References in curriculum, readings or 
teaching materials within education 

Bibliometric analysis 

Logs of information requests maintained by researchers 
Qualitative case studies (direct contact with those who can 
validate use of research): interviews and/or surveys of key 
informants 


First developed in Stanwick, Hargreaves and Beddie (2009; updated as at January 2015). 

*AltMetrics are metrics and qualitative data that are complementary to traditional, citation-based metrics. They can include (but are not limited to) citations on the web and in public policy documents, discussions on research blogs, 
mainstream media coverage, and mentions on social networks such as Twitter. 



^ Scope of this study 


A total of 32 reports were included in this impact study (see tables 4 and 10) across two 
themes. A mix of NCVER researchers and researchers funded under the National Vocational 
Education and Training Research program are represented. An attempt has been made to 
include a wide-ranging depiction of the research available under each theme, noting 
however that this is by far not the complete collection of research available. 

The two themes differ in specificity, in breadth and in timing of research output and 
measurable outcomes. This selection was deliberate, the aim being to test and evaluate 
different parts of an overall research portfolio. Theme 2 in particular is a more recent body 
of work. 

Theme 1 : The role of apprenticeships in a modern economy 

Apprenticeships are the cornerstone of the vocational education and training system and 
form an enduring aspect of both the research and statistics divisions of NCVER. A particular 
focus and body of work exists around completion, which as a ‘hot topic’, engages all major 
stakeholders - industry, providers and government, as well as individual apprentices. This 
case study covers the period 2005 to 2014 with a mix of research and statistics, as well as a 
major consultancy project for the Commonwealth Government included for consideration. 

Theme 2: The nature of vocations and the competencies 
required by industry 

In 2010 NCVER funded several research consortia to conduct significant bodies of work under 
major themes of the national research priorities 2011-2014 . One of those consortia 
comprised researchers from the LH Martin Institute, University of Melbourne, and the 
Workplace Research Centre, University of Sydney. Their research program ‘Vocations: the 
link between post-compulsory education and the labour market’ resulted in numerous 
research publications. This collection of theoretical work explores the notion of using 
vocations as a means of improving pathways within education, within work and between 
education and work. The publication dates are quite recent (in measuring impact terms) 
and many of the seminal dissemination activities likely to influence impact are still 
occurring (as at the time of this impact report). The purpose of including the work of the 
consortia is to set a benchmark which will feed into a longer-term impact study on this 
theme. 

There are connections between the research undertaken by the consortium and other 
research in this broader theme, particularly around the way in which vocational educational 
institutions function and the relationship between qualifications and the labour market. 

This vocations work also touches on the implementation of competencies and training 
packages. Hence, the addition of some other NCVER publications under this broad theme, 
including work on defining and interpreting competencies in VET, responding to changing 
skill demands, pathways from education to work, the principles on which VET is based, and 
qualification utilisation. 
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Not everything that 
counts can be 
counted, and not 
everything that can 
be counted counts 
(Cameron 1963). 


Methodology 

Citation analysis 

The bibliometric analysis was conducted by specialist information services staff using 
Google, Google Scholar, Publish or Perish, parliamentary databases and VOCEDplus. 2 The 
time period was from each report publication date to 25 May 2015. While comprehensive, a 
citation analysis is a point-in-time activity; it is not a true reflection of the level of actual 
engagement and use of the research. Our previous work in this area suggests the peak time 
for academic citations is around five to six years following report publication. The complete 
results of the citation and media analysis are provided in the support document 
accompanying this report. 


Google analytics 

Included in the metrics is the collection and analysis of data from the NCVER website/portal 
and the VOCEDplus database, using Google analytics. It tracks how visitors interact with the 
websites, including how many visit, where they come from, who they are (in terms of 
NCVER-defined user groups), how long they spend on the website, what pages they visit and 
what publications they view or download. Detailed Google analytics and methodological 
caveats regarding the reporting period, sampling, member group scope and tracking code 
issues in relation to the transition from NCVER’s website to the portal (which occurred on 3 
February 2014) are included in the support document. 


Editorial Board evaluations 

For this study we trialled the use of NCVER’s Editorial Board as an external assessment 
panel, adapting the case study evaluation questions and matrices from the United Kingdom 
Research Excellence Framework (REF) exercise. Each editorial board member was allocated 
between two and four reports to review. Evaluations were conducted over the period April 
to June 2015. A copy of the evaluation template is provided in the support document. 


Stakeholder questionnaire 

End-user stakeholders were randomly identified from a list of current subscribers to NCVER, 
from within categories of people potentially interested in the topics. This list was 
supplemented with individuals or organisations likely to have a general interest in the VET 
sector. The stakeholders invited to participate for each theme are identified in table 2, 
along with the responses received. In addition, ten authors of the reports under 
consideration were invited to participate and eight completed the questionnaire. A list of 
those who consented to have their name identified is provided in appendix A and the 
questionnaire instrument is provided in the support document. 


2 VOCEDplus is NCVER’s free international research database for tertiary education. It is international in scope 
and contains over 65000 English language records, many with links to full text documents: 
<http://www.voced.edu.au/>. 
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Limitations in the sample 

In terms of the scale of this study the response rates to the surveys for both themes are 
satisfactory and sufficient for meaningful analysis (table 2). However, the responses are not 
representative and do not reflect the broad range of NCVER’s stakeholder base. Response 
bias is also evident in the over-representation of industry affiliation or peak bodies under 
the apprenticeships theme. In addition, there were only two responses from a 
state/territory government and no responses from the Commonwealth Government. As a 
result, the analysis should be interpreted with discretion. 


Table 2 Stakeholders invited to participate and survey response rates 



Number of 

organisations 

contacted 

Number of 
individuals 
contacted 

Number of 
responses 
from 

individuals 

Theme 1 Apprenticeships 

Policy 

Commonwealth Government 

1 

3 

0 

State/Territory government 

5 

8 

2 

Industry 

Skills councils, licensing authorities and 

7 

7 

1 

employers 

Affiliation or peak body 

2 

6 

5 

Training fund organisations 

3 

3 

0 

Chambers of commerce and industry 

2 

3 

0 

Group training/apprenticeship centre or broker 

7 

15 

3 

Education 

Affiliation or peak body 

1 

1 

0 

Training providers (including colleges for 

4 

6 

3 

school based apprentices, and students) 

Union 

1 

1 

0 

Other research or statistical organisation 

1 

1 

0 

Did not identify 

n/a 

n/a 

1 

Total theme 1 

Response rate 

34 

54 

15 

27.8% 

Theme 2 Vocational competencies 

Policy 

Commonwealth Government department 

1 

2 

0 

State/Territory government department 

3 

4 

0 

Other non-profit or independent association 

8 

8 

1 

Industry 

Skills councils and employers 

8 

8 

0 

Affiliation or peak body 

8 

8 

2 

Chambers of commerce and industry 

4 

4 

0 

Education 

Affiliation or peak body 

5 

5 

2 

Training providers (including VET in Schools) 

4 

4 

0 

Did not identify 

n/a 

n/a 

2 

Total theme 2 

Response rate 

41 

43 

7 

16.3% 
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‘All knowledge is 
valuable, and having 
a broader base 
understanding and 
underpinning 
knowledge gained 
from engaging with 
the research 
supports my current 
and previous training 
roles’ (Training 
provider). 


© Findings: theme 1 - 
apprenticeships 

Introduction 

The analysis in this section is presented under the four main categories of the impact 
framework: knowledge production, capacity-building, informing policy and informing 
practice. The reporting period for this analysis was from the date of report publication to 
25 May 2015. The discussion incorporates the results of the citation analysis as well as 
survey responses, noting again the sample is not representative of all of NCVER’s 
stakeholders. Full results are provided in the support document. 


Knowledge production 

General engagement and use of the research 

Understanding the various ways in which stakeholders engage with research is important. 
We therefore asked them to identify, in general and broadly, how or why they use research 
or data on apprenticeships. Their verbatim responses are shown in table 3. The individual 
replies clearly demonstrate the diverse range and contexts in which NCVER reports function 
as a key source of information and evidence. The responses were wide-ranging, so the 
overarching categories of ‘understanding’, ‘developing’ and ‘communicating’ are used as 
these appear to best capture the high-level functions of engagement. 


Table 3 Stakeholder verbatim responses to the general capacity in which NCVER 
apprenticeships data and research are used 


Understanding 

Developing 

Communicating 

analysing key industry trends 

improving training programs 

liaising with and informing 

for better use and 

and services offered 

stakeholders 

interpretation of learner data 

designing training packages 

informing and formulating 
policy positions 

expanding and applying 

strategic planning and 


knowledge to personal 

monitoring 

progressing, lobbying, 

interests 


supporting or advocating 


preparing key papers, such 

arguments (including in 

maintaining currency of VET 

as business plans, workforce 
development plans or 

industrial tribunals) 

comparing with other 

environmental scans 

informing students and 

jurisdictions, previous years 


parents, and assisting with 

and industry sectors in terms 

writing ministerial 

decisions about future 

of apprentice commencements 
and completions 

submissions or briefings 

apprenticeship development 

directions of the college 
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Table 4 Apprenticeships theme - report details and overall metrics, from publication date to May 2015 


# 


1 


2 


3 


4 


Report 


Authors Publication Research purpose 

Stakeholder 

Media - 

Citations 

Downloads 

date 

awareness 

NCVER 

(including 

from NCVER 


of specific 

metrics 

some 

website and 


report 


media) 

portal (also 


(n = 15) 



see figure 2) 



Apprentice and 
trainee completion 
rates 


Katrina Ball 2005 

and David 

John 



The impact of wages 
on the probability of 
completing an 
apprenticeship or 
traineeship 


Tom 

Karmel and 
Peter 

Mlotkowski 


2010 


I 


How reasons for not 
completing 
apprenticeships and 
traineeships change 
with duration 


Tom 2010 

Karmel and 
Peter 

Mlotkowski 



Experimental 
completion and 
attrition rates for 
latest commencing 
apprentices and 
trainees 


NCVER 

statistical 

report 


2010 


This publication reports 
completion rates for 
apprentices and trainees 
who commenced their 
apprenticeship or traineeship 
between 1995 and 1999, and 
attrition rates for more recent 
commencements. 

The primary focus of this 
research is to examine the 
impact of wages on the 
decision not to continue with 
an apprenticeship or 
traineeship. The approach 
taken is to model three 
wages relevant to 
apprentices and trainees: the 
wage during training; the 
expected wage in alternative 
employment; and the 
expected wage on 
completion. 

This occasional paper adds 
to the considerable literature 
on low completion rates for 
apprenticeships and 
traineeships by looking at 
whether the reasons given 
for not completing vary by 
how far the individual is into 
their contract of training. 

This paper presents 
experimental estimates of 
completion and attrition rates 
by selected occupations for 
apprentices and trainees 
commencing in the 
December quarters of 2007 
to 2009. These estimates are 
supported by a technical 
paper which details the 
methodological approach 
taken to derive such up-do- 
date estimates. 


12 2 77 


9 31 25 


11 1 15 


5 1 11 


650 


1011 


592 


443 


Full record 
visits on 
VOCEDplus 


Australian 

Policy 

Online 

downloads 


Australian 

Policy 

Online 

hits 


1 88 n/a n/a 


22 23 2653 


29 47 3544 


45 n/a n/a 





Attrition in the trades Tom 

ha 

Karmel, 

liK_: > 

Patrick Lim 


and Josie 


Misko 


6 


■ Effect of the 
downturn on 
apprentices and 
trainees 


Tom 201 1 

Karmel and 
Damian 
Oliver 


7 


8 




The impact of wages 
and the likelihood of 
employment on the 
probability of 
completing an 
apprenticeship or 
traineeship 

Pre-apprenticeship 
training activity 


Tom 

Karmel and 
Peter 

Mlotkowski 


Paul Foley 
and 

Davinia 

Blomberg 


2011 


2011 


9 



Pre-apprenticeships 
and their impact on 
apprenticeship 
completion and 
satisfaction 


Tom 201 1 

Karmel and 
Damian 
Oliver 


In this paper, using 
professional occupations as 
comparators, the authors 
investigate attrition in the 
trades and the extent to 
which this is affected by 
good or bad economic times. 
They also ponder the 
question of whether a trade 
is a good start to a career by 
tracking the occupational 
destination of those who exit 
their trades. 

This paper looks at 
apprentice and trainee data 
and employment data back 
to 1995 and finds that 
declines in apprentice 
commencements lead to 
declines in total trades 
employment by four 
quarters, whereas declines 
in trainee commencements 
appear to lag declines in 
employment in non-trade 
occupations. 

This occasional paper looks 
at how the probability of 
getting a job, in addition to 
wages, impacts on 
completion rates. 


This research investigated 
pre-apprenticeship training in 
Australia through a variety of 
research methods: keyword 
searches, lists of prescribed 
pre-apprenticeship courses 
from state and territory 
contacts, and a unit record 
enrolment file. 

Pre-apprenticeship programs 
have attracted a great 
degree of interest from 
employers and training 
providers as one means of 
increasing apprentice 
satisfaction and completion 
rates, by providing an 
introduction to the trade. This 
proposition is tested using 
data from the 2010 
Apprentice and Trainee 
Destination Survey. 


11 9 9 


115 8 


9 8 15 


5 3 7 


8 15 22 


854 


570 


699 


678 


2148 


32 32 2144 


29 n/a n/a 


77 30 2187 


42 n/a n/a 


273 34 3945 
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Individual-based Tom 2011 Low completion rates for 5 13 20 1763 47 25 2897 

completion rates for Karmel apprentices and trainees 

apprentices have received considerable 

attention recently and it has 
been argued that NCVER 
seriously understates 
completion rates. In this 
paper, the author uses 
NCVER data on 
recommencements to 
estimate individual-based 
completion rates. 


Completion and 

NCVER 

2011-14 

11-1: 2010 

10 

50 

53 

927 

195 

n/a 

2461 

attrition rates for 

standard 


11-2: 2011 




2038 

210 

28 

2266 

apprentices and 

statistical 


11-3: 2012 




1879 

58 

68 

2786 

trainees - annual 

report 


11-4: 2013 




913 

30 

n/a 

n/a 


statistical 

publications 
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NCVER research 

Various 

2011 

Report 1 

10 

n/a 

53 

n/a 

91 

n/a 

n/a 


reports for 

NCVER 

Consultancy 

Report 2 





55 




Apprenticeships for 

authors 

project for 

Report 3 





34 




the 21st Century 


the 

Report 4 





131 




Expert Panel: 

Report 1: Overview 
of the Australian 
apprenticeship and 
traineeship system 
Report 2: Overview 
of apprenticeship 
and traineeship 
institutional 
structure 
Report 3: The 
apprenticeship and 
traineeship system's 
relationships with 
the regulatory 
environment 


Common- 

wealth 

Government. 

Reports 

were 

published by 
the 

Common- 
wealth 
department 
linked to the 
final report of 
the expert 
panel. 


Report 4: The 
economics of 
apprenticeships and 
traineeships 


13 



Understanding the 
psychological 
contract in 
apprenticeships and 
traineeships to 
improve retention 


Erica 201 1 

Smith, 

Arlene 
Walker and 
Ros 

Brennan 

Kemmis 


This paper compares the 
perceptions of Australian 
apprentices/trainees and 
employers of 
apprentices/trainees in 
relation to the psychological 
contract: the unwritten 
mutual expectations 
employers and employees 
have of each other. 


8 5 13 1524 32 n/a n/a 





14 


Evolution of Brian 2012 

apprenticeships and Knight 

traineeships in 
Australia: an 
unfinished history 


15 


The role of ‘culture’ 
in apprenticeship 
completions 


Tom 2012 

Karmel and 
David 
Roberts 


16 



Understanding the 
non-completion of 
apprentices 


Alice 

Bednarz 


2014 


Totals apprenticeships theme 


This occasional paper traces 
the evolution of Australia’s 
apprenticeship system from 
European settlement in 1788 
to the present day, including 
the various policies, social 
changes, and skill needs that 
have influenced how the 
system developed into what 
is in place today. 

This paper documents and 
finalises some work 
undertaken for the 
Apprenticeships for the 21 st 
Century Expert Panel. It 
aims to explain the extent to 
which variation in 
apprenticeship completion 
rates can be attributed to 
factors relating to the 
‘culture’ of the employer and 
apprentice. ‘Culture’, in this 
case, refers to employer 
size, type (government, 
private or group training) and 
intensity of trades workers in 
the apprentice’s home 
suburb. 

Approximately half of all 
apprenticeship contracts in 
the trades are not 
completed. This literature 
review draws together 
existing research and data to 
find out why apprentices do 
not complete their training. 


7 1 18 1784 169 5371 


16 


2087 


199 


n/a 


9 17 11 1112 216 139 


n/a 167 373 21 672 2 204 n/a 


Notes: Media coverage provides a general indication of a topic resonating across a broader audience, and there are a number of contributing factors that determine what may ‘grab’ attention at a particular time. There is minor 
duplication between some media counted in the citations column and NCVER media metrics. 

Metrics for Understanding the non-completion of apprentices will not be a good indication as yet because of the publication date; it has been included as a benchmark for future impact studies. 


3077 


n/a 


1593 


n/a 






The overall metrics on each report are provided in table 4. Of those who responded to the 
survey, all indicated they were familiar with the research and statistical reports published 
by NCVER, while the number of respondents familiar with one or more of the specific 
reports under consideration was relatively high. These results suggest consistently 
reasonable awareness amongst stakeholders of the specific publications available in the 
area of apprenticeships. 

An attempt was made to determine if respondents typically use information from the 
specific reports listed or whether they rely on a combination of publications from NCVER 
and other resources. While most of those who responded will generally combine various 
NCVER publications with other resources, half indicated they do use the specific findings 
from a particular NCVER publication. 

The report respondents were most familiar with was the oldest - Apprentice and trainee 
completion rates by Katrina Ball and David John, published in 2005. While now over ten 
years old, this remains one of the seminal reports on the topic. 

In line with our impact philosophy we recognise the importance of disseminating knowledge 
via a range of media, including conferences and presentations. Half of the respondents 
recalled attending one of NCVER’s presentations on research related to apprenticeships in 
general, supporting the broader transmission of knowledge in this area. 

Some of NCVER’s publications are available via the Australian Policy Online (APO) site. The 
metrics available on downloads and hits from the APO are included in table 4 as an 
indication of the research becoming more widely accessible and known beyond NCVER’s 
usual stakeholder base: APO has a subscriber readership of 17 000, 46% of which is from the 
government sector. 3 

The citations found refer not only to the traditional research report, conference paper or 
journal article but also ministerial press releases, submissions to parliamentary inquiries and 
references as far-reaching as Adult Learning Inspectorate reports from the United Kingdom. 
More detail on the evidence from the citations about impact on policy and practice is 
explored below, while the full details from the citation analysis can be found in the support 
document. 

Top downloads 

Figure 2 indicates the total number of downloads by report. The report with the largest 
number of downloads was Pre-apprenticeships and their impact on apprenticeship 
completion and satisfaction. 


3 <http://apo.org.au/about>. 
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Figure 2 Role of apprenticeships in a modern economy: total number of downloads (website and 
portal) by report from publish date 


2500 



4 5 6 7 8 9 10 11-1 11-211-311-4 12 13 

Report number: see table four for title and further detail 


Notes: Some statistical reports have been archived from the NCVER website to VOCEDplus; therefore, the number of 
downloads for these are from publication date to archive date. Report 12 was not published by NCVER; it 
therefore has no metrics available. 

Source: NCVER collated Google analytics data. 


What reports are resonating with stakeholders 

Figure 3 shows the extent to which the overall apprenticeships theme resonates with key 
member groups (as measured by downloads). Government is the highest user, followed by, 
in relatively equal proportions, research organisations or universities, TAFE (technical and 
further education) institutes or colleges, industry and private or other non-TAFE training 
providers. Schools, union and media outlets are relatively modest users. 

Figure 3 Role of apprenticeships in a modern economy: member group downloads 

2500 
2000 
1500 
1000 
500 
0 

Government Other Research TAFE Industry Private or School Union 
organisation institute or other non- 

or university college TAFE 

training 

provider 

Notes: Data are from the website only on 14 reports. As the website was de-commissioned on 3 February 2014, 
reports 11-4 and 16 were published on the portal only. 

Source: NCVER collated Google analytics data. 
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Table 5 identifies the reports that resonated most with a selection of key stakeholder 
groups. 


Table 5 Top three reports that resonated (and were downloaded) by select key member groups: 
from the website or portal (theme 1) 


Stakeholder 

Report 

number 

Report title 

Total number 
of downloads 

Government 

10 

Individual-based completion rates for apprentices 

278 


14 

Evolution of apprenticeships and traineeships in 

Australia: an unfinished history 

240 


16 

Understanding the non-completion of apprentices 

215 

Training 

16 

Understanding the non-completion of apprentices 

273 

organisation/provider 

10 

Individual-based completion rates for apprentices 

229 


11-3 

Completion and attrition rates for apprentices and 
trainees 2012 

213 

Industry 

11-2 

Completion and attrition rates for apprentices and 
trainees 201 1 

121 


10 

Individual-based completion rates for apprentices 

107 


11-3 

Completion and attrition rates for apprentices and 
trainees 2012 

101 

Media 

2 

The impact of wages on the probability of completing an 
apprenticeship or traineeship 

20 


11-3 

Completion and attrition rates for apprentices and 
trainees 2012 

18 


11-2 

Completion and attrition rates for apprentices and 
trainees 201 1 

17 


International profile 

Across all of the reports there have been international downloads from the United Kingdom, 
United States, Canada, New Zealand, Switzerland, Kenya, Greece, South Korea, Thailand, 
Italy, Indonesia, Germany, Estonia, France, India, Austria and China. It is interesting to note 
the report that received the highest number of international downloads - from the United 
Kingdom - was Understanding the psychological contract in apprenticeships and 
traineeships to improve retention. This may be reflective of the high profile and networks 
generated by the lead author, Erica Smith, in that country. 

Sharing research with others 

Almost all of the respondents indicated they had shared NCVER’s reports with others. The 
reasons provided for sharing reports were diverse: 

■ undertaking monthly reporting to demonstrate business performance 

■ providing information and advice to member associations and the business community 
generally, or as one respondent eloquently described as ‘educating and illuminating’ 

■ drawing attention to data and for comparing different datasets 

■ developing funding proposals 

■ supporting broader understanding of apprenticeships, with the aim of leading to 
improvements in apprentice support 

■ getting further advice about interpreting the research findings and the potential for 
influencing practice 


NCVER 
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The ability of NCVER 
to supply industry 
with data should be 
more widely 
promoted. The data 
we received 
dedicated to our 
industry has been a 
major influence on 
industry policy and 
planning’ (Industry 
peak body). 


■ providing the opportunity for others who may have an interest in the research to read 
the full report 

■ informing industry bodies and business of the state of play in the industry. 

Capacity-building 

Most of the respondents said the research had a developmental or other educational benefit 
either for themselves, within their own organisation or for others. Just under half indicated 
their engagement with the research led them to make use of other NCVER research or data. 

Various types of capacity-building arising from engagement with the research were 
reported: 


■ assisting in the development of specific programs and initiatives 

■ understanding market trends and conditions, for example, the problems associated with 
non-completion of apprenticeships, the downturn in apprentice commencements and 
general issues concerning apprentice and trainee outcomes 

■ the data providing an excellent benchmark to use for commencement, completion and 
in-training figures 

■ supporting a program for encouraging school-based apprentices 

■ assessing the trends in behaviours against the services and support provided to 
apprentices 

■ assisting in planning recruitment strategies and training delivery processes. 


Informing policy 

Evidence from stakeholders 

All respondents indicated they thought research overall leads to better information by 
which to inform or influence policy in the VET sector. More specifically, stakeholders were 
also asked if they were aware of the research leading to any observable changes in policy or 
having influenced government programs or other initiatives. Nine of the 15 respondents 
indicated in the positive. 

The following quotes give an indication of how research is used in the policy process by non- 
government stakeholders. The views highlight the role of independent research efforts. 

They also point to the degree to which conflicting knowledge can be used ‘to construct 
claims that challenge established patterns of influence’ (Daviter 2015, p.499). 

It can certainly influence the way that one presents one’s arguments, particularly if 
research findings do not necessarily support a particular position for which one is 
arguing. It can also trigger thinking about further research topics or questions for 
investigation. (Industry affiliation or peak body) 

It makes it so much easier to develop policy when you can refer to relevant research to 

support a position. In addition if we need to talk to the Minister about an issue we can 

say ‘but the research says ...’ (Industry affiliation or peak body) 
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Respondents were asked whether they have used the knowledge gained from the research in 
any way to inform policy. Six of the 15 respondents indicated possibly but they could not 
provide direct evidence. Eight could provide details (table 6). 


Table 6 Stakeholder evidence of impact on policy: apprenticeships theme 


Examples or quotes provided by stakeholders 
regarding the use of research in policy 

Stakeholder 

Specific reports mentioned (if 
applicable) Refer to table 4 for 
referencing of report number 

Research has informed initiatives to improve completion rates. 
Specific example: research undertaken for the former NSW 
Board of Vocational Education and Training (BVET). 

State/territory 

government 

department 

1,2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 15, 

16 

Funding rates for apprentices and trainees in remote locations 
of the Northern Territory were increased. 

Incentives for group training organisations were increased for 
remote Indigenous communities to assist with 
commencements, progressions and completions, which has 
helped to increase the numbers. 

State/territory 

government 

department 

1,3, 5, 6, 12 

Research has been used for expansion planning, and is 
regularly referred to in government and stakeholder 
engagement. Specific documentary evidence: 

Pricewaterhouse Coopers’ review of the National 
Apprenticeships Program (NAP), using NCVER research. 

Group training 
organisation 

1,2, 3, 4, 5, 6, 11, 12, 13, 14, 15, 

16 

Apprentice completion rates data influenced the 
implementation of a mentoring program. 

Apprenticeship 
centre or broker 

N/A 

The research commissioned by the Apprenticeships for the 

21st Century Panel led to the formulation of policy that had a 
disastrous impact on apprenticeship commencements in 
Australia. 

Industry affiliation 
or peak body 

1,2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 14, 
15, 16 

To some extent NCVER statistics and research underpinned 
the mentoring and support policies and programs and the new 
Apprenticeship Support Network. 

Specific documentary evidence within various Australian 
Chamber of Commerce and Industry publications: A 
systematic approach to retaining apprentices', It’s not about 
age ...; Worth their weight in gold ; From apprenticeship to a 
career. 



Policy and programs associated with pre-apprenticeships. 
Initiatives associated with improving apprentice 
commencements and completions; work in partnership with 
the Queensland and Northern Territory governments. 

Industry affiliation 
or peak body 

1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 

14, 16 

We are always using the NCVER work on completion rates 
and reasons for non-completion to make certain points in the 
hope of influencing policy. 1 am not sure that there is a direct 
link but 1 think one could argue that NCVER’s and other work 
has influenced the government decision to have Australian 
Apprenticeship Support Network providers focus more on 
activities designed to improve completion rates. 

1 also think that a lot of the NCVER work on VET in Schools is 
providing momentum for governments to keep questioning the 
purpose and improving the quality of these programs. 

1 think separate papers by Phil Toner and Mark Cully on 
traineeships over the years, commissioned by various parties, 
have influenced government thinking about the value of 
traineeships, the extent to which they should attract 
government subsidies and the contribution they make to 
national skills formation. 

Industry affiliation 
or peak body 

1,2, 3, 5, 6, 7, 9, 11, 12, 13 

Generally, about skill shortages, support services, funding 
allocation and the provision of training. 

Commencements and completions often influence the skills 
shortage data, in particular the funding of apprenticeship 
training and the skilled occupation list. This data influences at 
the state and federal level. 

Industry affiliation 
or peak body 

All 

Development of an industry five-year training strategy. 

Industry affiliation 
or peak body 

1, 3, 11 

Has led to greater support for apprentices, especially during 
their first year. 

Secondary college 

12 
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Informing practices associated with the employer Apprentice N/A 

management of apprentices and informing potential 

apprentices. 

Even when respondents could not provide specific examples they made reference to links 
between the research and government policies or guidelines on the implementation of 
apprenticeships and traineeships. As one respondent cautioned however the data on 
apprenticeships alone do not reflect the complexity of the apprenticeships system. 

I suspect NCVER data collection for both group training organisations and VET in 
general has significantly influenced the ‘bean counters’ in Canberra. As I have often 
stated however NCVER statistics do not capture the effort involved in employing 
apprentices. (Industry affiliation or peak body) 

Of note are the connections various stakeholders are making between research and 
subsequent new policy and funding related to apprenticeship mentoring and pastoral care 
(table 6). 4 While it is not possible to prove direct causation, the introduction of the 
Australian Apprenticeships Mentoring Package in 2011, which recognised the importance of 
pastoral care arrangements for apprentices to improve completion rates, came in the wake 
of research undertaken by NCVER and others. 

Conflicting views exist regarding the use of research to inform policy, as reflected in the 
comments included in table 6. Depending on who the stakeholder is and their role within 
the sector they, as would be expected, will perceive connections between the research and 
the consequences of subsequent policy shifts differently. They may also be directly 
affected, either positively or adversely, through decisions arising from policy. 

This is most evident in the research reports underpinning the review of Australian 
Apprenticeships for the 21st Century Expert Panel. As an example, the intention of NCVER’s 
report The economics of apprenticeships and traineeships was to bring together some of the 
more important economic considerations such as evidence on costs and benefits, as well as 
sustainability issues that might impact on the long-term future of the apprenticeship and 
traineeship system. Four ideas were suggested for the deliberations of the expert panel: 

• In occupations where the apprenticeship/traineeship model has a near 
monopoly in entry level training, reduce the risk of having an inadequate 
labour supply by supplementing the model with an institution-based one, 
perhaps with some sort of provisional qualification that is completed with 
work experience. 

• Look at the possibility of abandoning the model in occupations where it is 
clearly not working. In these cases, an apprenticeship or traineeship would be 
replaced by an institutional path. 

• Question the merits of having traineeships for existing workers and part-time 
workers. Many of these seem to be about reducing wage costs to employers 
and getting a government subsidy rather than skills acquisition. On-the-job 
training will provide the necessary skills for most of these workers. 


4 For a collated history of key policy shifts related to apprenticeships and traineeships from 1939 to 2015, see 
<http://www.voced.edu.au/content/timelines>. 
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• Being clear about which policies are concerned with skills acquisition and 
which are concerned with assisting entry into the labour market. Perhaps 
government incentives should go to individuals rather than occupations. 

(Karmel a Rice 2011, p.7) 

The expert panel subsequently recommended that the Australian Government ‘redirect 
employer incentives to provide structured support services to eligible apprentices and 
trainees and their employers in occupations that are priorities for the Australian economy’ 
(Apprenticeships for the 21st Century Expert Panel 2011, p.4). The government then 
undertook further analysis by commissioning an econometric review of the Australian 
Apprentices Incentives Program (Deloitte Access Economics, 2012). The various reviews and 
analyses provided evidence and argument that led to decisions regarding the redesign of the 
overarching policy related to incentives. From 2012 there have been reductions in the 
incentives paid to employers, which appear to have had a flow-on effect, mainly in reduced 
commencements of non-trade apprenticeships and traineeships (Atkinson & Stanwick 
forthcoming). 

This example emphasises the complex interplay of evidence and subsequent modification to 
an existing policy or incremental change. Examining the outcomes of the review of the 
Australian Apprenticeships for the 21st Century Expert Panel also reveals that not all the 
research entered into the ensuing policy deliberations. This suggests that other 
considerations affect the policy-makers’ decisions, including the views of stakeholders with 
a vested interest in the outcomes, as well as the capacity of the government to act. 

Overall, the recommendations that were acted upon were concerned with refinements to 
the existing system and many of the fundamental issues the evidence raised were not 
addressed. 


Evidence from the citation analysis 

A number of reports from this theme have been used in policy submissions and the 
formulation of legislation on apprenticeships in Australia and internationally, with key 
highlights from the citation analysis reported in table 7. 


Table 7 Citation evidence of impact on policy: apprenticeships theme 


Policy or submission 
reference 

Source(s) 

Reference 

date 

Reports used 

Policy/legislation 

Skills Account: no TAFE fees 
for traditional trades 
apprentices 

Australian Labor Party, 
Parliament of Australia 

2006 

Apprentice and trainee completion 
rates 

Support for trade apprentices 

Julia Gillard and the 
Australian Labor Party 

2010 

Experimental completion and attrition 
rates for latest commencing 
apprentices and trainees 

Democratic Alliance policy on 
labour (South Africa) 

Democratic Alliance, 

Cape Town, South 

Africa 

2013 

Report 4: The economics of 
apprenticeships and traineeships 

Senate Inquiry into the Trade 
Support Loans Bill 2014 and 
the Trade Support Loans 
(Consequential Amendments) 
Bill 2014 

Australian Council of 
Trade Unions 

Champion, N 

Ey, C (personal 
submission) 

2014 

Individual based completion rates for 
apprentices 

Understanding the non-completion of 
apprentices 

Industry innovation and 
competitiveness agenda: an 

Department of the Prime 

2014 

Australian vocational education and 
training statistics: completion and 
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action plan for a stronger 
Australia 

Minister and Cabinet 


attrition rates 

for apprentices and trainees 2012 
(published 2013) 

Submissions 

Victorian Government Inquiry 
into Vocational Education and 
Training 

Australian Education 
Union Victorian Branch 

2005 

Apprentice and trainee completion 
rates 

Independent Pricing and 
Regulatory Tribunal, NSW 
Review of the Skills Base in 
NSW and the Future 
Challenges for Vocational 
Education and Training 

NCVER 

NSW Department of 
Education and Training 

2006 

Apprentice and trainee completion 
rates 

Fair Work Australia (matter 
no.C201 1/1) 

Australian Business 
Industrial and NSW 
Business Chamber 

2010 

The impact of wages on the probability 
of completing an apprenticeship or 
traineeship 

Skills Australia: Creating a 
future direction for Australian 
vocational education and 
training 

Housing Industry 
Association 

NSW Business 

Chamber 

2010 

The impact of wages on the probability 
of completing an apprenticeship or 
traineeship 

Experimental completion and attrition 


rates for latest commencing 
apprentices and trainees 


Smart and Skilled: making NSW State Training 201 1 

NSW number one Services 


A shared responsibility: 
Apprenticeships for the 21 st 
Century Expert Panel Report 
(stakeholder submissions) 


Australian Chamber 
Alliance 
AVETRA 
AC PET 

Minerals Council of 
Australia 

Smith, E (personal 
submission) 

EE-OZ Training 
Standards 


2011-2012 


2012 Skilled Occupations List Auto Skills Australia 2012 

(AW PA) 


Modern Award Apprenticeship Australian Federation of 


Wage Review 201 2 - 
AM2012/18 and others, 
apprentice, trainees and 
juniors (Fair Work 
Commission) 


Employers and 
Industries 

Australian Council of 
Trade Unions 
Australian Chamber of 
Commerce and Industry 
Toner, P (personal 
submission) 

Australian Mines and 
Metals Association 


2012-2013 


Report 2: Overview of apprenticeship 
and traineeship institutional structures 
Report 3: The apprenticeship and 
traineeship system’s relationships with 
the regulatory environment 

The impact of wages on the probability 
of completing an apprenticeship or 
traineeship 

How reasons for not completing 
apprenticeships or traineeships 
change with duration 
Experimental completion and attrition 
rates for latest commencing 
apprentices and trainees 
Pre-apprenticeships and their impact 
on apprenticeship completion and 
satisfaction 

Understanding the psychological 
contract in apprenticeships and 
traineeships 

Report 1: Overview of the Australian 
apprenticeship and traineeship system 
Report 2: Overview of apprenticeship 
and traineeship institutional structures 
Report 4: The economics of 
apprenticeships and traineeships 

Australian vocational education and 
training statistics: completion and 
attrition rates for apprentices and 
trainees 2010 (published 201 1 ) 

Apprentice and trainee completion 
rates 

How reasons for not completing 
apprenticeships or traineeships 
change with duration 
The impact of wages and the 
likelihood of employment on the 
probability of completing an 
apprenticeship or traineeship 
Individual based completion rates for 
apprentices 

Australian vocational education and 
training statistics: completion and 
attrition rates for apprentices and 
trainees 201 1 (published 201 2) 
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Evolution of apprenticeships and 
traineeships in Australia: an unfinished 
history 

The role of ‘culture’ in apprenticeship 
completions 

Report 1: Overview of the Australian 
apprenticeship and traineeship system 
Report 2: Overview of apprenticeship 
and traineeship institutional structures 
Report 4: The economics of 
apprenticeships and traineeships 


Changes to Indigenous 
Employment Program 
assessment and funding 
processes (discussion paper) 

Complete Personnel 
Group 

2013 

Australian vocational education and 
training statistics: completion and 
attrition rates for apprentices and 
trainees 2010 (published 201 1 ) 

Productivity Commission 
inquiry into geographic labour 
mobility 

NCVER 

2013 

Attrition in the trades 

House of Representatives 
Standing Committee on 
Education and Employment 
Inquiry into the Role of TAFE 
and its operation 

Holmesqlen Institute of 
TAFE 

2014 

Evolution of apprenticeships and 
traineeships in Australia: an unfinished 
history 

Productivity Commission's 
Inquiry into Workplace 

Relations Framework (Case 
study: sham contracting in the 
construction industry) 

Australian Council of 
Trade Unions 

2015 

The role of ‘culture’ in apprenticeship 
completions 

Senate Standing Committees 
on Economics on the 
Privatisation of State and 
Territory Assets and New 
Infrastructure 

The Australian 

Municipal, 

Administrative, Clerical 
and Services Union 

2015 

Understanding the non-completion of 
apprentices 


Informing practice 

Evidence from stakeholders 

Most respondents thought the apprenticeships research had led to general improvements in 
practice, either for themselves, within their organisation or more broadly within the VET 
sector (figure 4). 

Figure 4 Stakeholder awareness of impact on practice: apprenticeships theme 



■ No, not aware 


■ Yes, for myself 


■ Yes, within my organisation 


Yes, more broadly within the 
vocational education and 
training sector 
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In terms of whether the research overall leads to better information by which to inform or 
influence practice in the VET sector, 13 of the 15 respondents said 'yes'. A reason given for 
why research may not lead to improvement in practice was that it is ‘too time stale’. 

The stakeholder evidence provided in table 8 tends to demonstrate more generic or 
anecdotal links and also that any change in practice related to apprenticeships, especially 
completions, occurs through wider support structures and systemic policy shifts. 


Table 8 Stakeholder evidence of impact on practice: apprenticeships theme 


Examples provided by stakeholders validating 
the use of research 

Stakeholder 

Specific reports mentioned 
(if applicable) 

Improved planning and implementation (commissioned 
reviews). 

Group 

training 

organisation 

1,2, 3, 4, 5, 6, 11, 12, 13, 14, 15, 

16 

Some research into completions has led to the adoption 
of better recruitment and pastoral care practices by 
employers, group training organisations and Australian 
Apprenticeship Centres. 

Industry 
affiliation or 
peak body 

1,2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 14, 

15, 16 

Research has provided the evidence to change policy 
and develop programs. 

Industry 
affiliation or 
peak body 

1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 

14, 16 

It is too soon to say whether the new service 
requirements imposed on Australian Apprenticeship 
Support Network contractors, which it is hoped will 

Industry 
affiliation or 
peak body 

1,2, 3, 5, 6, 7, 9, 11, 12, 13 


improve completion rates, have achieved anything 
because they did not come into effect until 1 July 2015. 
The fact they have been introduced though is something 
of a victory for research and advocacy. 

The work done by Rod McDonald and Mary Dickie for 
the former NSW Board of Vocational Education and 
Training encapsulated in the report A fair deal (and 
referencing several NCVER research reports) has 
provided some of the evidence for mismatching of 
prospective apprentices to employers and the need for 
recruiters to apply better matching techniques. This 
work has been operationalised in the development of a 
website <www.employingapprentices.com.au>, which it 
is hoped will prove to be a useful tool for various 
intermediaries recruiting people for apprenticeships or 
directing prospective apprentices towards 
apprenticeships such as career advisers and VET in 
Schools coordinators. 


Changes to training delivery and recruitment practices; 
trade journal stories and industry newsletters. 

Industry 
affiliation or 
peak body 

1, 3, 11 

There were a number of guides, such as good practice 
in training and assessing, which are probably dated now 
but were extremely useful at the time of print. 

Training 

provider 

1,2, 3, 5, 6, 7, 8, 9, 13, 14, 15, 16 
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Evidence from the citation analysis 


There is also some evidence from the citation analysis of research from this theme 
informing practice, reported in table 9. This takes the form of new publications aimed at 
practitioners. 


Table 9 Citation evidence of impact on practice: apprenticeships theme 


Publication type 

Source(s) 

Reference 

date 

Reports used 

A guide to the employment of 
apprentices and trainees in the 

NSW public sector 

NSW State Training 
Services/Public Sector 
Industry Training Advisory 
Body 

2008 

Apprentice and trainee 
completion rates 

Hospitality management: strategy 
and operations 

Book chapter in Business 
relationships 

2012 

Experimental 
completion and 
attrition rates for latest 
commencing 
apprentices and 
trainees 
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Findings: theme 2 - vocations 


Introduction 


The analysis in this section is presented under the four main categories of the impact 
framework: knowledge production, capacity-building, informing policy and informing 
practice. The reporting period for this analysis was the date of report publication to 25 May 
2015. The discussion incorporates the results of the citation analysis as well as survey 
responses, noting again the sample is not representative of all of NCVER’s stakeholders. Full 
results are provided in the support document. 


Additional limitations for this theme 


There were not as many respondents to this theme, despite the survey being distributed to 
a similar number of stakeholders as the apprenticeships theme (54 individuals for theme 1 
and 43 individuals for theme 2). This may in part be attributed to timing: the first report 
was published in 2012. We are aware the citation rates will not yet paint a comprehensive 
picture. In addition, the fact sheets accompanying the research reports and a series of 
stakeholder events on this work occurring between 2014 to 2016 are not included in the 
metrics. Therefore, the consortia publications have been included as a benchmark for 
future impact studies, which will also take into account the feedback from attendees who 
attended presentations on this work. 


Knowledge production 

General engagement and use of the research 

All respondents to the survey indicated they are familiar with NCVER’s publications. The 
metrics on each report and the number of respondents familiar with specific reports under 
consideration are identified in table 10: they reveal a low level of awareness. 

The majority of respondents indicated they turn to various NCVER publications, as well as 
resources outside NCVER, when seeking out information. A few respondents recalled 
attending a presentation on research related to this theme. 
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Table 10 Vocations theme - report details and overall metrics, from publication date to May 2015 


# Report 

Authors 

Publication Research purpose 

Stakeholder 

Media - 

Citations 

Downloads 

Full record 

Australian 

Australian 



date 

awareness 

NCVER 

(including 

from NCVER 

visits on 

Policy 

Policy 




of specific 

metrics 

some 

website and 

VOCEDplus 

Online 

Online 




report 


media) 

portal (also 


downloads 

hits 




ii 

c 



see figure 5) 





Research consortia publications from the three year (2011-14) consortia research program: Vocations: the link between post-compulsory education and the labour market 


Entry to vocations: Kira Clarke 

current policy and Veronica 

trends, barriers and Volkoff 

facilitators of quality 
in VET in Schools 


201 2 This working paper sets the 

policy context and direction of 
the entry to vocations research 
stream. After looking at the 
current 'in-school' element of 
VET in Schools programs and 
jurisdictional differences, the 
paper raises a set of questions 
for discussion. For example, 
are more advanced and intense 
programs of VET in Schools 
needed to deliver stronger 
labour market outcomes? Do 
VET in Schools programs need 
to be more directly connected 
with post-school VET 
programs? 



2 0 7 


625 


80 


n/a n/a 


2 


3 


4 




Entry to vocations: Kira Clarke 2012 This report explores the 

the efficacy of VET relationship between vocational 

in Schools education and training in VET 

in Schools and the labour 
market. 


Missing links: the 
fragmented 
relationship between 
tertiary education 
and jobs 

Leesa 
Wheelahan, 
Mary Leahy, 
Nick 

Fredman, 

Gavin 

Moodie, 

Sophie 

Arkoudis and 

Emmaline 

Bexley 

2012 

This report explores the 
transitions students make in 
undertaking a second 
qualification (i.e. whether they 
change field of education 
and/or move between VET and 
higher education sectors). It 
also looks at the reasons why 
they decide to undertake 
another qualification. 

Revitalising the 

Leesa 

2012 

This discussion paper 

‘vocational’ in flows 

Wheelahan, 


introduces the overarching 

of learning and 

Gavin Moodie 


concepts for the three-year 

labour 

and John 


research program, ‘Vocations: 


Buchanan 


the link between post 


compulsory education and the 
labour market’. The paper also 
outlines the key findings from 
the initial investigations of each 


0 1 32 1 998 72 n/a n/a 


17 


1726 


60 


30 


1866 


2 0 24 598 49 n/a n/a 
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The role of 
educational 
institutions in 
fostering vocations 


Gavin Moodie 2012 


6 


Understanding the 
nature of vocations 
today: exploring 
labour market 
pathways 


Serena Yu, 2012 

Tanya 

Bretherton, 

Johanna 
Schutz and 
John 

Buchanan 





Vocational 
trajectories within 
the Australian labour 
market 


Serena Yu, 2012 

Tanya 

Bretherton 

and Johanna 

Schutz 


of the related research strands 
in the program. Overall, initial 
research has found that, due to 
the fragmentation of pathways 
within education and within the 
labour market, the relationship 
of pathways between education 
and the labour market are also 
fragmented. In order to 
understand this more clearly, 
the authors explore the 
concepts of transition systems, 
skill ecosystems and 
capabilities. A number of 
dilemmas for further research 
are also put forward. 

This working paper is an initial 
examination of what is known 
about pathways in tertiary 
education, as well as the loose 
associations between 
vocational qualifications and 
the jobs graduates do. 

Much research into the reasons 
students undertake tertiary 
study is at least implicitly based 
on rational choice theory: 
students calculate costs in 
order to maximize direct 
individual gain. The policy 
emphasis on pathways has 
somewhat broader bases 
including social inclusion as 
well as economic productivity, 
but narrowly focuses on 
institutional arrangements 
within education to facilitate 
upward transfer within fields of 
education. This paper critiques 
such assumptions. 

This report explores the 
movements that workers make 
in the workforce and whether 
there are any commonalities. It 
also considers whether these 
movements can be 
characterised as vocational 
pathways. Workers within the 
finance, primary, health and 
electrical trades/engineering 
industries were interviewed 
about their employment and 
study history, career 
progression and reasons for 
any movements. 


0 0 13 296 27 n/a n/a 


0 0 12 


551 


35 n/a n/a 


13 


986 


47 


n/a 


n/a 
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Defining vocational 
streams: insights 
from the 
engineering, 
finance, agriculture 
and care sectors 


Serena Yu, 
Tanya 
Bretherton 
and John 
Buchanan 


2013 


9 


10 



Entry to vocations: Kira Clarke 2013 

strengthening VET 
in Schools 


Vocational 
education’s variable 
links to vocations 


Gavin 2013 

Moodie, Nick 
Fredman, 

Emmaline 
Bexley and 
Leesa 
Wheelahan 


The authors identify the 
potential for vocational streams 
- a set of linked occupations 
with related skills in a broad 
field of practice - in the areas 
of agriculture, financial 
services, engineering and 
healthcare/community 
services. 

How VET in Schools can be 
strengthened to provide 
stronger links to post-school 
pathways is explored in this 
report. 

Using NCVER and Australian 
Bureau of Statistics data, as 
well as findings from four case 
studies, this report looks at how 
mid-level qualifications assist 
entry to and progression in the 
workforce. 


1 2 11 1249 32 n/a n/a 


1 6 9 1488 61 n/a n/a 


1 2 17 970 41 n/a n/a 


11 


Entry to vocations: 
building the 
foundations for 
successful 
transitions 


Kira Clarke 


2014 


Improving the outcomes of 
students undertaking vocational 
education and training in 
schools is important to ensuring 
it is a viable post-school 
pathway for students. 
Investigating how to strengthen 
VET in Schools, this report 
suggests the following 
changes: defining its purpose 
as foundational preparation for 
mid-level qualifications or 
employment-based learning; 
better aligning the vocational 
curriculum with labour market 
opportunities; developing 
thematic packages of 
curriculum; and making explicit 
connections between VET 
undertaken at school and post- 
school VET. 


0 


14 


591 


26 n/a n/a 


Other NCVER publications of relevance 


12 





Responding to 
changing skill 
demands: training 
packages and 
accredited courses 


Josie Misko 


2010 


This report looks at whether 
vocational education and 
training is equipped to meet the 
changing needs of the modern 
workplace. 


12 22 2437 38 26 3108 









From education to 
employment: how 
long does it take? 


Darcy 201 1 

Fitzpatrick, 

Laurence 
Lester, Kostas 
Mavroamaras, 

Sue 

Richardson 
and Yan Sun 


14 



VET research for 
industry 


Tom Karmel 2012 

(conference 

paper) 


15 



Steven Hodge 2014 

competencies in 
Australian vocational 
education and 
training: practices 
and issues 


16 


Bridget 2014 

utilisation: Wibrow 

occupational 
outcomes - 
overview 


Using the 1995 cohort of the 
Longitudinal Surveys of 
Australian Youth (LSAY), this 
project examines how long it 
takes for young people to move 
from education to work, and 
identifies factors that 
distinguish people who are able 
to move quickly into 
employment from those who 
take many months to find work. 

This paper was a keynote 
address at the Australian 
Vocational Education and 
Training Research Association 
(AVETRA) conference held in 
2012. The author notes that 
industry is arguably the key 
stakeholder in the Australian 
VET sector and discusses six 
areas of research which raise 
questions about VET and 
industry, either in terms of how 
public training funds are 
allocated or the educational 
principles on which VET is 
based. 

How VET practitioners 
understand and use 
competency standards is of 
fundamental importance to the 
quality and integrity of the 
Australian VET system. This 
small study seeks to address 
this question by gaining insights 
from 30 VET practitioners. 

This overview uses information 
from the Student Outcomes 
Survey (SOS) to match the 
intended occupations of 
graduates with their destination 
occupation six months after 
training. The influence of 
qualification level, labour force 
status before training, age, 
completing only a module and 
industry area was also 
examined. 


0 14 23 1579 41 n/a n/a 


4 


11 


2288 


26 n/a n/a 


2 2 17 2886 89 n/a n/a 


0 2 17 


613 


24 n/a n/a 


Total vocations theme 


n/a 50 259 20 881 748 n/a n/a 








Top downloads 

Figure 5 indicates the total number of downloads by report. The report with the largest 
number of downloads was Interpreting competencies in Australian vocational education and 
training: practices and issues. This may be a reflection of both the importance and appetite 
for research on this issue, as well as the title immediately making sense to stakeholders. 

Figure 5 Theme 2: nature of vocations - total number of downloads (website and portal) by 
report from publish date 
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Report number: see table ten for title and further detail 

Source: NCVER-collated Google analytics data. 


What reports are resonating with stakeholders 

Figure 6 shows the extent to which the overall reports in this theme are being downloaded 
amongst key member groups. As with theme 1, government is the highest user. 


Figure 6 Theme 2: nature vocations - member group downloads 
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Table 11 identifies the reports that resonated most with a selection of key stakeholder 
groups. 


Table 11 Top three reports that resonated (and were downloaded) by select key member groups: 
from the website or portal (theme 2) 


Stakeholder 

Report 

number 

Report title 

Total number 
of downloads 

Government 

12 

Responding to changing skill demand: training 
packages and accredited courses 

221 


2 

Entry to vocations: the efficacy of VET in Schools 

219 


16 

Qualification utilisation: occupational outcomes - 
overview 

189 

Training 

organisation/provider 

15 

Interpreting competencies in Australian vocational 
education and training: practices and issues 

356 


12 

Responding to changing skill demand: training 
packages and accredited courses 

330 


14 

VET research for industry 

257 

Industry 

12 

Responding to changing skill demand: training 
packages and accredited courses 

64 


15 

Interpreting competencies in Australian vocational 
education and training: practices and issues 

60 


2 

Entry to vocations: the efficacy of VET in Schools 

54 

Media 

13 

From education to employment: how long does it take? 

7 


14 

VET research for industry 

7 


8 

Defining vocational streams: insights from the 
engineering, finance, agriculture and aged care sectors 

6 


International profile 


Reports across this theme are being downloaded in the following countries: the United 
Kingdom, United States, Canada, New Zealand, Italy, Indonesia, Germany, China, South 
Africa, Trinidad and Tobago, Vietnam, Chile, Netherlands, Hong Kong and the Philippines. 

The report with the highest download internationally was Interpreting competencies in 
Australian vocational education and training: practices and issues , from the Philippines. 
While Missing links: the fragmented relationship between tertiary education and jobs had 
particular interest from the United Kingdom. 

Sharing research with others 

Five of the seven respondents indicated they have shared NCVER’s research with others, 
predominantly via social media or newsletters. The main purposes for sharing reports 
include: 

■ extending the knowledge base 

■ broadening understanding of the VET landscape 

■ sharing key findings with colleagues, consultants or networks. 
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Capacity-building 

Almost half of the respondents were aware of the research having had a developmental or 
other educational benefit either for themselves, within their own organisation or for others. 
Various types of capacity-building arising from engagement with the research were 
reported, such as: 

■ informing the activities of the organisation 

■ increasing the knowledge of staff and colleagues 

■ applying it to practitioner research 

■ applying for and receiving a research grant. 

Around half indicated their engagement with the research had or probably had led them to 
make use of other NCVER research or data. 


Informing policy 

All respondents indicated they thought research overall leads to better information to 
inform or influence policy in the VET sector. Stakeholders were also asked if they were 
aware of the research in this theme generally leading to any observable changes in policy or 
influencing government programs or other initiatives in any way, with half of respondents 
indicating 'yes', even though there was less awareness of the specific reports under 
consideration in this case study. 

Respondents were also asked whether they had used the knowledge gained from the 
research in any way to inform policy. The responses were evenly spread, with two 
respondents indicating ‘no', another two indicating ‘possibly’, but they are unable to 
provide direct evidence and two respondents indicating ‘yes’ with the ability to provide 
details (see table 12). In addition, the researchers involved in the consortia work reported 
follow-on policy-relevant research directly arising from the work undertaken for NCVER by 
the Board of Vocational Education and Training in NSW. 


Table 12 Stakeholder evidence of impact on policy: vocations theme 


Examples provided by stakeholders validating Stakeholder Specific reports mentioned 
the use of research (if applicable) 

Industry Skills Responding to changing skill 

Council demands: training packages 

and accredited courses 
‘VET research for industry’ 
(conference paper) 

Entry to vocations: current 
policy trends, barriers and 
facilitators of quality in VET in 
Schools 

Revitalising the ‘vocational’ in 
flows of learning and labour 
Vocational trajectories within 
the Australian labour market 
Interpreting competencies in 
Australian vocational 
education and training: 
practices and issues 

We always try to ensure policy development is Industry peak N/A 

evidence-based. NCVER research was used in the body 

development of the National Agriculture Workforce 
Development Plan. The adoption of skill sets as part of 

government-funded educational outcomes. We have 


The research is used heavily in submissions to 
Commonwealth and state/territory government 
departments. 

Greater awareness of the role of VET and of the key 
workforce training issues facing Australia. We have 
been asked to present at Senate hearing and House of 
Representatives enquiries. 

In the community services and health industry our work 
has definitely informed policy. The Industry Skill 
Councils are mandated to provide analysis of key trends 
to policy, with NCVER research cited in most ISC 
environment scans. 


NCVER 


41 



pushed for skill set funding in a number of policy 
proposals; the research was able to provide an evidence 
base supporting this approach. 5 This is just one 
example of a suite of reforms affecting the VET funding 
model. 

Evidence from the citation analysis 



A number of reports from this theme have been used in various submissions and legislation 
relating to policy, identified in table 13. Of note is policy related to the review of training 

packages and accredited courses. 



Table 13 Citation evidence of impact on policy: vocations theme 


Policy or submission 
reference 

Source(s) 

Reference 

date 

Reports used 

Policy/Legislation 

Social Security Legislation 
Amendment (Family 
Participation Measures) 

Bill 201 1 , Second Reading 

House of 

Representatives, 

Hansard 

2011 

From education to employment: 
how long does it take? 

Submissions 

Standing References 
Committee on Education, 
Employment and 

Workplace Relations - 
Inquiry into Industry Skills 
Councils 

NCVER 

2010 

Responding to changing skill 
demands: training packages and 
accredited courses 

Productivity Commission 
Study on the Schools 
Education and Training 
Workforce 

Australian Education 

Union 

2011 

From education to employment: 
how long does it take? 

Australia’s skills and 
workforce development 
needs (discussion paper) 

Australian Industry 

Group 

2012 

Entry to vocations: strengthening 
VET in Schools 

Employment services - 
building on success 

Butler, L (no other 
source identified) 

2013 

Entry to vocations: current policy 
trends, barriers and facilitators of 
quality in VET in Schools 

House of Representatives 
Standing Committee on 
Education and 

Employment - Inquiry into 
the Role of TAFE and its 
Operation 

AgriFood Skills Australia 
Manufacturing Skills 
Australia 

Unions WA 

2013 

Entry to vocations: the efficacy 
of VET in Schools 

Entry to vocations: strengthening 
VET in Schools 

Inquiry into Vocational 
Education and Youth 
Training Programs in the 
ACT 

NCVER 

2013 

Entry to vocations: the efficacy 
of VET in Schools 

Making the Grade: 
Autonomy and 
Accountability in Victorian 
Schools: Inquiry into 

School Devolution and 
Accountability 

VET Development 

Centre 

2013 

Entry to vocations: the efficacy 
of VET in Schools 

Productivity Commission 
Inquiry into Geographic 
Labour Mobility 

NCVER 

2013 

Vocational trajectories within the 
Australian labour market 

Review of Australian 
Curriculum 

Australian Industry 

Group 

2013 

Entry to vocations: the efficacy 
of VET in Schools 


5 The research this refers to did not explicitly form part of this impact project; it is J Mills, D Crean, D 
Ranshaw 8t K Bowman, Workforce skills development and engagement in training through skill sets, NCVER, 
Adelaide 2012. Available at <http://www.ncver.edu.au/publications/2568.html>. 
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Review of the demand 
driven funding system 

Monash University 

2013 

Defining vocational streams: 
insights from the engineering, 
finance, agriculture and care 
sectors 

Submission to the 

Victorian Government for 
the 2014-15 budget 

Australian Industry 

Group 

2013 

Entry to vocations: strengthening 
VET in Schools 

Review of current HSC 

VET work placement 
arrangements in NSW 

NSW Business Chamber 

2014 

Entry to vocations: the efficacy 
of VET in Schools 

Review of training 
packages and accredited 
courses 

Department of Industry 

2014 

Responding to changing skill 
demands: training packages and 
accredited courses 

Qualification utilisation: 
occupational outcomes - 
overview 

The Forrest Review - 
Indigenous Jobs and 
Training (Employment 
Review Taskforce, 
Department of the Prime 
Minister and Cabinet) 

NCVER 

2014 

Qualification utilisation: 
occupational outcomes - 
overview 

Productivity Commission 
Inquiry into the Workplace 
Relations Framework 

Buchanan, J (one of the 
authors of the research) 

2015 

Revitalising the ‘vocational’ in 
flows of learning and labour 
Defining vocational streams: 
insights from the engineering, 
finance, agriculture and care 
sectors 

The operation, regulation 
and funding of private 
vocational education and 
training providers in 
Australia 

Melbourne Graduate 
School of Education, 
University of Melbourne 

2015 

Revitalising the ‘vocational’ in 
flows of learning and labour 

Entry to vocations: building the 
foundations for successful 
transitions 

Submission to the 

Victorian Government for 
the 2015-16 budget 

Australian Industry 

Group 

2015 

Entry to vocations: strengthening 
VET in Schools 


Informing practice 

Most respondents thought the research had exerted some impact on practice within their 
organisation or more broadly within the VET sector. See figure 7 and table 14. 


Figure 7 Stakeholder awareness of impact on practice - vocations theme 


■ No, not aware 

■ Yes, within my organisation 

■ Yes, more broadly within the 
VET sector 
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‘I would suggest that 
without the source 
material I was able to 
use from NCVER 
there would be a 
large void for VET in 
this country’ 
(Education peak 
body). 


Even though the data come from a very small number of responses, to some extent the 
findings with respect to informing practice in this theme are surprising, given the largely 
theoretical nature of the research and especially given the early stages of this analysis 
following publication of the consortia work. 

Stakeholders themselves recognised that this theme was not necessarily directed at practice 
or practical matters, as it ‘concerns really large structural issues in VET’. Another 
respondent suggested that when talking about change in practice it was more about looking 
for research that supported a particular outcome and it was really only in government that 
this research could support actual change. 

Herein lies a major issue with interpreting impact, which might be described as a 
‘gestation’ factor. This is the weight of evidence and information assembled over a period 
of time that, only as a collective body of work, leads to potential or significant policy 
change, which typically is in the final hands of government. Theme 2 examines and in part 
challenges current policy in vocational training products and training pathways with a 
forward future perspective of skill needs in a changing labour market. Being a large 
‘structural issue’, it is rightfully contested ground with differing views. Where policy or 
practice changes are deemed worthy, this will take time. 


Table 14 Stakeholder evidence of impact on practice: vocations theme 


Examples provided by stakeholders validating Stakeholder Specific reports mentioned (if 
the use of research applicable) 


Research on efficacy of mentoring has influenced the VET 
development of mentoring support attached to a Development 

grants program Centre 

Information regarding VET in Schools has influenced 
the professional development provided to VET in 
Schools teachers 

Research grants of $5000 each were provided to 
support VET practitioners to develop research 
capability 

The organisation more broadly is mindful of 
contemporary practice arising from the research; for 
example, with their work on behalf of the Victorian 
Government, support of AVETRA and management 
of a blog providing national and international 
information related to vocational education and 
training. 

Professional program in work-integrated learning for Education 
TAFE Queensland, including learning guides, peak body 

webinar series and supporting material 

Used NCVER research information in the design and 
development of a professional development program 
for TAFE teachers in work integrated learning 


VET research for industry 
(conference paper) 

Revitalising the ‘vocational’ 
flows of learning and labour 

Entry to vocations, 
strengthening VET in Schools 

Vocational education’s variable 
links to vocations 


VET research for industry 

Defining vocational streams: 
insights from the engineering, 
finance, agriculture and care 
sectors 

Interpreting competencies in 
Australian vocational education 
and training: practices and 
issues 
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Evidence from the citation analysis 

Keeping in mind the relatively recent publication dates and the overall purpose of 
developing theoretical frameworks and influencing policy thinking for most of the research 
considered in this theme, there is already evidence from the citation analysis of some 
research informing practice, as follows (table 15): 


Table 15 Citation evidence of impact on practice: vocations theme 


Publication type 

Source(s) 

Reference 

date 

Reports used 

Career education 
benchmarks - 
secondary: a set of 
quality benchmarks for 
career education 
programs and services 
in New Zealand 
secondary schools 

Careers New Zealand 

2014 

Entry to vocations: 
strengthening VET in Schools 

Train the Trainer - 
ASEAN national 
assessor trainee 
manual 

William Angliss 

Institute of TAFE 

2015 

Responding to changing skill 
demands: training packages 
and accredited courses 
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1 The views of researchers 


The views of researchers are important, particularly in terms of the capacity-building 
domain. While professional or career development benefits may arise from the conduct of 
research, we are also interested in collaborations formed with end-users and the 
researchers’ own perspective of the influence they are having on policy and practice. 
Another benefit of engagement, which NCVER plays a key role in facilitating, is a better 
appreciation of the policy environment and how researchers can best engage with it. 
Including the views of researchers is a useful exercise for uncovering how research can add 
value in ways other than usually conceived. Our findings are reported against the four 
domains of our framework. 

Knowledge production 

All researchers who responded to the survey reported they involved key stakeholders in the 
design and conduct of their research. In addition, most attended specific stakeholder 
meetings or presentations as well as participated in a wide range of other dissemination 
activities, such as media interviews, webinars and social media, and a variety of 
conferences. 

While most researchers were aware of their research being publicised in some way, they 
were generally less aware of their research impacting on policy or practice. They 
acknowledged the challenge of capturing impact and recognised that once they moved on 
from the research project they were less likely to be aware of the impact of their research. 

Many did say they spent a great deal of time talking to people in government about their 
work and this alone had increased awareness of the research. In addition, where possible, 
researchers were involved in the presentation of early research findings to key stakeholders 
and invited them to offer appraisals and suggestions for dealing with the issues raised. 

It is my experience as an author of NCVER reports that stakeholders follow what NCVER 
releases very closely and accord its findings a great deal of weight. Much more weight 
than, say, reports that stakeholders themselves may have commissioned from 
consultants. During conferences I would get very detailed questions from attendees 
about how the results related to their industry and whether or not they accorded with 
their own experiences. (Researcher) 

Impact on policy 

While some researchers were able to identify impacts on policy, many acknowledged they 
did not keep track of what happens to their research once published. One researcher 
commented on the limited scope for impact when the outcomes of one report alone do not 
yield meaningful results: 

Potentially, the research could have had a much greater impact but the results were a 

bit of a wash. If there had been an overwhelming finding one way or another, then I 

think there would have been much more of a reaction from policy-makers. (Researcher) 
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Others had a more positive story to tell. Some researchers from the vocations consortia had 
direct involvement in policy development and saw how their research was being used: 

■ by the South Australian Training Board to think about policy at the state level 

■ at a ‘think session’ in Tasmania with TAFE, the government department and key 
stakeholders 

■ as the basis for the Australian Curriculum, Assessment and Reporting Authority’s 
(ACARA) National Trade Cadetships initiative, with the researchers being asked by 
ACARA to develop the curriculum framework. 

Impact on practice 

Researchers who have a teaching role in either a university or vocational provider indicated 
they often use the results from their research in their own teaching programs, and they 
encourage their colleagues to do the same. 

Capacity-building 

Most researchers were aware of their research generating subsequent research in the area, 
conducted either by themselves or by others. At least half benefited in terms of attracting 
further research funding. Only a small number of researchers were able to identify a 
promotion or secondment as a result of their research track record. However, for one 
researcher, a sequence of projects (both NCVER and other work) was instrumental in their 
being appointed as Personal Chair in VET within their university and had overall played a 
part in their recognition as a leading authority globally in apprenticeships research. 

Two examples are worth noting with respect to capacity-building. They highlight the value 
of the consortia approach for bringing researchers together and opening up networks and 
opportunities to others on the team. 

Understanding the psychological contract in apprenticeships and 
traineeships to improve retention 

Researchers who worked on this project reported that it enabled them to participate in an 
international research project for the World Bank and the International Labour 
Organization’s Indian offices. It also informed international work on apprenticeships, 
particularly with the International Network on Innovative Apprenticeship (INAP) network. As 
mentioned earlier, the lead researcher was able to promote and highlight this research in 
the United Kingdom via conference presentations and existing networks. 

Vocations: the link between post-compulsory education and the labour 
market 

The vocations research consortium established the foundation for further research proposals 
sent to various government departments and organisations interested in the capabilities 
approach to education and training. A lead researcher was successful in obtaining several 
further research grants, including two projects for the Board of Vocational Education and 
Training in NSW. In addition, most researchers indicated that their involvement in the 
consortium led to the development of new connections with other researchers, enabling a 
better understanding of issues pertaining to VET. Ongoing collaborations with the broader 
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VET community and other stakeholders, including those responsible for implementing 
policy, were also established. As one researcher stated: ‘my network of expert colleagues 
has expanded meaningfully during the project’ . The lead researcher and others in this 
consortium have presented the findings at many conferences, and continue to do so. On the 
basis of this work, the researchers were able to maintain research connections in Australia 
and build research connections in the United Kingdom and the United States through an 
international network of researchers working on ‘higher vocational education’ pathways. 
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o The Editorial Board analysis 

NCVER places top priority on producing high-quality research and statistical publications. An 
extensive range of quality assurance mechanisms guide the entire research process, from 
selection, with stakeholder input of research topics, the adoption of rigorous methodologies 
and data checking, to blind peer review and sign-off. We also require reports to be written 
in plain English, working directly with researchers and editors on this aspect. 

For this project NCVER’s Editorial Board has been asked to contribute to the impact 
assessment. The intention was to determine the extent to which our impact model could be 
enhanced, with the use of an independent judgment of the research, beyond metrics and 
the self-reported use of research by the stakeholders in our case studies. Members were 
asked to rate reports according to quality of output, potential impact and general 
assessment of accessibility. 

Underpinning the Editorial Board assessments was NCVER’s impact philosophy - and this 
philosophy perhaps also best defines our definition of quality: Research to have impact must 
be seen as relevant by its target audiences , make a useful contribution to knowledge , be 
impartial , adopt rigorous methodologies and use reliable data, be conducted ethically, be 
above criticism of bias, be interesting, and be read or at least become known. 

Editorial Board members were each asked to review between two and four reports, with 
some reports assessed by two members. Editorial Board members were paid, in accordance 
with NCVER’s standard peer review process. 

Editorial Board members involved in the analysis 

The following Editorial Board members were available to participate in the review process. 6 
Their biographies are included at appendix B. 

■ Professor Gerald Burke, Adjunct Professor, Faculty of Education, Monash University 

■ Professor David Finegold, Chief Executive Officer, American Flonors Organisation 

■ Dr Robin Ryan, Adjunct Lecturer, Graduate Program in Educational Leadership and 
Management, School of Education, Flinders University 

■ Professor Kwong Lee Dow, AO, Professor, University of Melbourne 

■ Dr Tom Karmel, Adjunct Professor, National Institute of Labour Studies, Flinders 
University. 

Reports selected for Editorial Board analysis 

The following 12 reports were included in the review, and those marked with an asterix 
received two reviews. Unfortunately, two other Editorial Board members were unable to 
complete the reviews of apprenticeship theme reports allocated to them; hence, the 
disparity in the number of reports reviewed across the two themes. 


6 To ensure anonymity and in accordance with privacy policies we will not divulge which Editorial Board 
member reviewed which report. Any conflicts of interest were declared. 
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Apprenticeships theme 

■ Attrition in the trades * 

■ Evolution of apprenticeships and traineeships in Australia: an unfinished history * 

■ Understanding the non-completion of apprentices 

■ Understanding the psychological contract in apprenticeships and traineeships to 
improve retention * 

Vocations theme 

■ Entry to vocations: building the foundations for successful transitions 

■ From education to employment : how long does it take ? 

■ Interpreting competencies in Australian vocational education and training : practices 
and issues * 

■ Missing links: the fragmented relationship between tertiary education and jobs * 

■ Qualification utilisation: occupational outcomes - overview * 

■ Responding to changing skill demands: training packages and accredited courses 
• Revitalising the ‘ vocational ’ in flows of learning and labour * 

■ Vocational education's variable links to vocations * 

Approach to analysis and feedback from the Editorial Board 

The criteria and rating scales were adapted from the Research Excellence Framework 
impact exercise, undertaken in the United Kingdom 7 , as follows. 

Quality of output 

Originality and rigour are the two criteria used for assessing quality of output. This takes 
into account the overall research design and methodology. Originality can be understood in 
terms of the innovative character of the research output. Research outputs that 
demonstrate originality may: engage with new and/or complex problems; develop 
innovative research methods, methodologies and analytical techniques; provide new 
empirical material; and/or advance theory or the analysis of doctrine, policy or practice. 
Rigour is understood in terms of the intellectual precision, robustness and appropriateness 
of the concepts, analyses, theories and methodologies deployed in the research output. 
Account will be taken of such qualities as the integrity, coherence and consistency of 
arguments and analyses, including the due consideration of ethical issues. 


7 NCVER acknowledges the use of assessment criteria from the Research Excellence Framework 2014. The 
copyright for that work is held by the Higher Education Funding Council for England (HEFCE). All content is 
available under the terms of the Open Government Licence v3.0 
<http://www.nationalarchives.gov.Uk/doc/open-government-licence/version/1/>. 
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Criteria 


1 star 


2 stars 


3 stars 4 stars 


Quality that falls 
below the 
standard of 
nationally 
recognised work, 
or work that does 
not meet the 
definition of 
research. 


Quality that is 
good overall 
but falls short 
of highest 
standards. 


Quality that 
is very good 
and 

recognised 

nationally. 


Quality that is 
of the highest 
standard and 
internationally 
excellent. 


Potential impact 

The criterion for assessing potential impact is ‘reach and importance’. Reach can be 
understood in terms of a professional judgment on the extent and diversity to which others 
(individuals, organisations, and communities) may benefit from the research. Importance 
can be understood in terms of the development of the intellectual agenda of the field and 
may be theoretical, methodological and/or substantive. Due weight should be given to 
potential, as well as actual significance, especially where the output is recent. The degree 
to which the research has the potential to influence or inform policy and practice should be 
taken into account. 


Criteria 1 star 

2 stars 

3 stars 

4 stars 

Impact is of little or no 

Considerable 

Very 

Outstanding 

reach or importance; or 

impact. 

considerable 

impact in 

the potential impact is 


impact. 

terms of 

not underpinned by 



potential 

excellent research. 



reach or 
importance. 


General assessment of accessibility 

NCVER’s foremost interest is in applied research; that is, research aimed at informing policy 
development or practice. NCVER therefore strives to produce research reports that are 
accessible to a wide range of stakeholders. They should be easy to read and interesting. 


Criteria 1 star 

2 stars 

3 stars 

4 stars 

Does not satisfy 

Partially satisfies 

Satisfies the 

Satisfies the criteria 

the criteria: 

the criteria: 

criteria: Supports 

to an exemplary 

Not interesting, is 

Further work is 

and adds new 

level: Major 

disjointed and 

required to add 

information, as a 

contribution, provides 

lacks coherency. 

value and ensure 

whole is coherent 

fresh insights, 

coherency. 

and logical. 

explores issues from 
different angles, high 
degree of coherency. 




Feedback on the criteria and rating scales 

Feedback from the Editorial Board suggested the rating process was mostly appropriate but 
there was room for improvement to suit NCVER’s remit. Members viewed it as a relatively 
straightforward exercise, although there was substantial work in reading, and in some cases 
re-reading, the reports, as well as undertaking additional work to familiarise themselves 
with the topic and broader literature if required. It was felt that international significance 
was an unrealistic measure for research so focused on specific Australian issues. This was 
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therefore interpreted as the research looking respectable in terms of methodology and 
execution if presented at an international conference. 

Some suggested there was a gap between three and four stars, especially in the first 
criterion for assessing the quality of output, so in some instances Editorial Board members 
used a 2.5 or 3.5 star rating. As one reviewer explained: 

The star system is well explained and should be useful for getting comparative 
impressions of component elements of the reports. I’d suggest care in not over 
interpreting the numbers which are provided. I tried to use the full scale for maximum 
differentiation, but with limited success. There wasn’t much that warranted four stars, 
though there were sections in all the reports that could have been given that grading. 
Especially in my reports 3 and 4 I found it hard to give one score for the whole report 
(on each of the dimensions) because of the good and less good parts of those quite 
extended documents. In such cases one averages, so there are lots of three’s. As one 
would expect, it would be misleading to give one star for any work at this level of 
sophistication with these chosen authors. So I reverted to invoking half stars, as they 
do in the hotel industry! This should help indicate some differentiation and seemed the 
best I could do. 

Editorial Board member reviews 

Table 16 combines all of the ratings allocated to provide, as an approximate guide, an 
overall average picture of quality, potential impact and accessibility. For reports that were 
reviewed by two members the average was used. 

Table 16 Editorial Board average ratings in relation to quality, potential impact and accessibility 
on a selected number of research reports 


Criteria 

Average 

Standard 


rating 

Deviation 


Quality of output 


la Originality 

2.56 

0.69 

1b Rigour 

2.79 

0.62 


Potential impact 


2a Reach 

2.71 

0.42 

2b Importance 

2.92 

0.47 


General assessment of accessibility 


3a Logical, well-structured and argued 

3.19 

0.53 

3b Written in clear, direct and plain English 

3.38 

0.36 


The general assessment of accessibility received the more consistent ratings from the 
Editorial Board overall and suggests that NCVER produces reports that are written in clear, 
direct and plain English. In terms of quality there is room for improvement in 
methodological rigour. There was also greater variance in the Editorial Board scores under 
the quality aspect but one reviewer also scored consistently lower than the others. 

The most revealing aspect of this exercise was the comments that most Editorial Board 
members submitted along with their rankings. These offered both positive remarks and 
constructive criticisms. A selection of each is provided below. 
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Positive remarks 

This report is an exemplary model of a clearly defined and well executed piece of 
educational research. It is modest in its ambit, and is careful in not over-claiming its 
importance or significance. The research question is elegantly presented, its relevance 
is explained with clarity and simplicity and its conclusion is authoritative and specific. 

So it won’t get lost in the welter of reports and issues which crowd in and complicate 
interpretation of key questions and possible solutions but should stand the test of time 
in offering a reliable ‘key message’ as formulated in the report. 

This is a very important study. It fills a gap in knowledge. The method of approach is 
innovative and the application excellent. As a ‘one off’ this can be regarded as an 
excellent study. I think the study needs to be seen as an element in an enormous body 
of work relating to apprentices and trainees much of it cited by the expert panel on 
apprentices and trainees. It is for this reason that I have ranked it at four stars on 
virtually all criteria. 

Criticisms 

With an array of knowledgeable and respected contributors this report could be 
expected to break new ground and make a substantial contribution to understanding in 
the field. It doesn’t quite live up to that and may disappoint in terms of its impact and 
in terms of take home messages that point a way forward. It may be that it tries to 
encompass too much, and with many authors perhaps compromises have been made so 
the report lacks a sharp and clear argument. It gets lost in its own analyses, needs 
rigorous pruning in places to avoid repetition, especially of ‘conclusions’. 

This report is very difficult to assess as a stand-alone research paper. I had difficulty 
finding a clear line of argument ... all in all the paper reads as a smorgasbord, with a 
great number of dishes. While some of the dishes are of interest there is no real feel of 
a well-structured dinner. The rigour of the report is hard to judge because it is a 
summary of other work. However, on the face of it the component pieces of research 
appear to be competently done. This report is unlikely to have a huge impact. 

Many of the comments pointed to the following challenges. These will inform NCVER’s 

efforts for continuous improvement in its management of research. 

■ being more critically aware of limitations in methodology and what could be done 
differently in research design at the beginning and early stages of the research cycle 

■ making sure there is balanced and wide consideration of the literature or data to back 
statements or findings, rather than relying on one or two sources to support a position 

■ ensuring reports have a clear line of argument, and ideas and results are well 
integrated 

■ checking and modifying titles of the research report as required to ensure they truly 
reflect the content (noting the metrics reported earlier reinforce the importance of 
titles) 
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And finally, as a couple of reviewers noted, there are other challenges affecting potential 
reach and impact: 

Impact affected by VET community’s lack of interest in history, not any deficiency in 
the research. 

The collected work by this team is a very important contribution though it seems 
unlikely to get as much attention as it deserves in the current political and 
administrative climate. 

Summing up the role of the Editorial Board 

This aspect of the impact evaluation added a layer of complexity to the exercise and gave 
rise to the following considerations. The first are the implications for the quality assurance 
already embedded in NCVER’s research and publication processes. The second relates to the 
extent to which we then explore, explicitly, aspects of quality post-publication as one 
element of our impact assessment. The exercise itself brings into question the 
appropriateness of rating individual reports as a short-cut to making an overall judgment 
about quality. After all, any impact exercise itself is implicitly judging quality; indeed, our 
impact philosophy, stated earlier, perhaps best defines what we mean by quality. 
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© General observations about 
dissemination 


Respondents to both themes were asked to rate how important, from least important to 
most important, particular types of research dissemination activities were in terms of 
facilitating engagement with the research. 

Across both themes there were 19 responses to this question. The results are presented in 
table 17. 


‘We would love to 
read the research in 
detail but in reality 
this does not happen, 
Of most value is the 
executive summary 
and snapshot of 
findings’ (Industry 
affiliation or peak 
body). 


Research findings translated to fact sheets, research overviews or good practice guides rank 
as the most important for encouraging engagement and use of the research, followed by 
reading the research report itself. Several respondents requested the development of more 
good practice guides. All recognise the importance of the research report itself as the 
underpinning evidence base and wish they had more time to read the research in detail. 

The findings regarding the importance of media and social media are interesting: almost all 
respondents indicated this approach to dissemination was not very important. This finding 
may be a reflection of a general misapprehension about the validity of the research 
messages reported via media and blogs for the purposes of developing policy and informing 
practice, and not a reflection of the importance of this type of approach for raising 
awareness. It may also be a reflection of demographics amongst the survey respondents. 
One respondent said: ‘I am in the older generation so social media does not work well for 
me’. 


Table 17 Importance of various types of dissemination products or activities for facilitating 
engagement with the research 


Type of dissemination or product 

Important 

Neither 

Not so 
important 

Research findings translated to fact sheets or research overviews 

17 

2 

0 

Research findings translated to good practice guides 

16 

3 

0 

Reading research reports 

14 

4 

1 

Participating on working groups or steering committees 

13 

4 

2 

Distributing the findings via newsletters 

11 

7 

1 

Attending seminars or conferences 

11 

7 

1 

Discussions with the researchers 

11 

4 

4 

Including the research in educational materials 

11 

4 

4 

Reading briefing papers 

10 

9 

0 

Reading journal articles 

9 

7 

3 

Access to research in progress working papers 

7 

7 

5 

Seeing media coverage 

6 

8 

5 

Engaging in social media (blogs/twitter) 

1 

8 

10 


In order of most important, n = 19*: n = 14 apprenticeships theme, n = 5 vocations theme. 
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9 Conclusion 

This study has assessed the impact of NCVER’s research against two themes: the role of 
apprenticeships in a modern economy; and the nature of vocations and the competencies 
required by industry. The evaluation spans 32 publications from 2005 to 2015. This is the 
third instance in which NCVER has undertaken an impact evaluation, on each occasion 
enhancing our approach. 

Despite the general caveats associated with assessing impact and our case study approach, 
the results are encouraging in terms of the type of impact NCVER is striving for. Indeed, the 
extent of evidence gathered from a relatively small base is surprising. 8 We also know our 
publications are reaching a sizeable number of stakeholders, both in Australia and 
internationally. In addition, our H-index score, a measure of academic impact, is a 
respectable 47 across all NCVER-authored publications. 

With respect to knowledge production across both themes in this particular study, we can 
identify: 

■ 42 553 downloads of the reports from NCVER’s website/ portal 

■ 2952 full record visits on VOCEDplus 

■ 632 citations across various sources, including policy and legislation 

■ 217 references across an assortment of media outlets. 

The actual use of the research is diverse, ranging from maintaining currency of knowledge 
to writing ministerial briefings. There is self-reported evidence of the use of research in 
both policy and practice terms, as well as substantial evidence of the research being used in 
policy documents and submissions. 

Within the apprenticeships theme, for example, there is evidence of the research 
influencing five official policy statements or legislation and numerous submissions across 13 
national or state/territory reviews. Most of those are government-related and at high levels, 
including House of Representative and Senate standing committees. 

There is stakeholder-reported evidence of the research being used to: 

■ influence initiatives to improve completion rates 

■ modify incentives for remote Indigenous communities to assist with commencement, 
progression and completion 

■ underpin mentoring and support policies and programs 

■ affect pre-apprenticeship and school-based apprenticeship arrangements 

■ inform the practices associated with employer recruitment, management and pastoral 
care of apprentices. 


8 The number of reports under consideration for this study was 32, bearing in mind that NCVER has published 
on average 42 research reports per year over the last decade. 
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Research under the vocations theme also received attention in one piece of official policy 
legislation and a number of submissions across 17 varied but primarily government-related 
inquiries. The use of research as reported by stakeholders highlighted the extent to which 
industry engages with the research in their attempts to influence government policy, with 
some success reported. For example, one industry used research on skills sets to support its 
case for funding, which did become available. Research in this theme is also driving several 
important VET practitioner professional development activities. Given the unpredictable 
ways in which research can have an impact and over a longer time period, we now have 
baseline metrics on which to revisit the impact of this work in a few years’ time. 

Recognising what we refer to as a ‘gestation’ factor is important. It is the weight of 
evidence and information assembled over a period of time and only as a collective body of 
work that has the potential to influence significant policy change - which typically is in the 
final hands of government. This is evident across theme 2 in particular, which challenges 
current policy in vocational training products and training pathways. Being a large 
‘structural issue’, it is rightfully contested ground characterised by differing views. 

This study has revealed awareness of specific NCVER publications, as well as an appreciation 
that a body of work on key themes can have substantial influence. We believe there is 
evidence of both knowledge creep, which is the research over time causing incremental 
policy steps or change on practices via existing policy structures (for example, with 
apprenticeships commencements). There is also evidence of knowledge shifts, whereby the 
research is playing a more direct role in observable changes to policy, such as that related 
to a focus on mentoring in attempts to lift completion rates. 

In many cases the researchers themselves are directly involved in publicising the findings, 
further extending awareness and knowledge, often directly with those involved in 
developing policy. The researchers in the vocations consortia in particular have played a 
central role in disseminating the findings and the results of this study recognise the 
importance of their continued efforts in this area. NCVER will continue to promote and 
support building the capacity of researchers to purposefully engage in the end use of their 
work. 

The impact of research is not easily predictable. Some stakeholders reported adverse 
consequences arising from either the research itself or ensuing policy decisions. This 
naturally influences their perception of the ways in which research and statistics may be, or 
in some cases may not have been, used in the development of policy. This is especially 
pertinent in the topical area of apprenticeship commencements and completions. 
Undertaking an impact assessment highlights both the indirect influence of research in the 
policy process, as well as how those who use the research can skew the way it is received. 
Thus: 

Any definition of research impact must take the long view of research influence into 
account as well as the contradictory or inconsistent views of users and reviewers. But 
research that generates new knowledge both inside and outside universities, research 
that can improve subsequent research, as well as research that influences the decisions 
(regardless of outcome) that shape people’s lives, communities, governance, the 
environment, and elsewhere can be defined as having impact. 

(Federation for the Humanities and Social Sciences 2014, p.9) 
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Regardless of the connections that end-users make, the stakeholders in this study revealed a 
high level of respect towards NCVER. They know they can turn to our publications as a 
trusted source of evidence to support or dismiss arguments. This in turn gives end-users the 
confidence to feed the information into policy deliberations. NCVER cannot influence how 
the evidence is used, although sophisticated stakeholder engagement and dissemination 
strategies can assist in knowledge transfer and uptake. 

We can also continue to provide information to the sector that adheres to our underpinning 
rigorous quality assurance principles. The Editorial Board’s assessment of the quality of 
some of the reports in this study was a sobering reminder that more can be done to improve 
rigour in research design and methodologies, as well as ensuring that arguments, ideas and 
findings are well integrated and flow logically in the write-up of the research. 

As with previous impact studies, NCVER’s flexible approach to dissemination of research and 
targeted communication strategies is affirmed as effective. While stakeholders recognise 
the importance of the research report itself, they have an appetite for shorter succinct 
products that offer the option of quick engagement with key findings from the research. 

This attests to the importance of our ‘About the research’ and key messages, as well as 
crisp executive summaries in reports. NCVER will continue to invest time and resources into 
those outputs, as well as into a variety of new products such as infographics and data 
visualisation. 

With respect to the impact framework, more needs to be done to harness non-traditional 
metrics (AltMetrics), currently only available if we plan in advance the collection of data 
prior to report publication. While social media may not factor as a trusted source of 
engagement with research to influence policy or practice, it does play an ever-increasing 
role in our dissemination strategies. It will be important to map the conversations that 
follow, especially via blogs or forums, and which may potentially translate into indirect 
influence. 

While the interplay between research and policy and practice is never straightforward, 
connections do occur. This report in particular has highlighted the remarkable variety in 
which NCVER publications are used, even amongst a small group of stakeholders. 

Understanding the impact of NCVER’s research is important not only to ensure we remain 
clearly focused on our remit to influence policy and practice in the training system, it also 
bolsters our collaboration with end-users of the research, helping us to produce the 
evidence in ways that best meet their needs. 
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0 Appendix A - respondents 


The following respondents to surveys consented to having their names identified in the 

report. 

Theme 1 

■ Ms Marie Anderson, Training Manager, The Apprentice and Traineeship Company 

■ Mr James Barron, CEO, Group Training Australia Ltd 

■ Mr Stephen Bolton, Senior Advisor Employment, Education and Training, Australian 
Chamber of Commerce and Industry 

■ Mr Patrick Gavaghan, National Training Manager, Australian Glass and Glazing 
Association 

■ Mr Wayne Lee, Industry Skills Adviser, Australian Industry Group 

■ Mr David Mitchell, Executive Officer, Apprentice Employment Network 

■ Mr Jeff Priday, National Projects Manager, Group Training Australia Ltd 

■ Mr Alan Sparks, CEO/Director, East Coast Apprenticeships 

Theme 2 

■ Dr Shayne Baker, OAM, President, Australian Institute of Work Based Learning 

■ Dr Brendan Goodger, National Manager Policy and Research, Community Services and 
Health Industry Skills Council 

■ Ms Sarah McKinnon, Manager Workplace Relations and Legal Affairs, National Farmers 
Federation 

■ Ms Denise Stevens, CEO, VET Development Centre 

Researchers 

■ Dr Steven Hodge, Lecturer, Griffith University 

■ Dr Gavin Moodie, Adjunct Professor, RMIT and University of Toronto 

■ Dr Damian Oliver, Deputy Director (Business Development), Centre for Management and 
Organisation Studies, University of Technology Sydney 

■ Professor Erica Smith, Personal Chair in VET, Federation University Australia 

■ Dr Arlene Walker, Associate Head of School (Rural and Regional Development), Deakin 
University 

■ Associate Professor Leesa Wheelahan, Department of Leadership Higher and Adult 
Education, Ontario Institute for Studies in Education, University of Toronto 
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Q Appendix B - Editorial Board 
member biographies 


■ Professor Gerald Burke, Adjunct Professor, Faculty of Education, Monash University 

From 2008 to 2013 Gerald Burke was a member of the board of Skills Australia 
and later the Australian Workforce and Productivity Agency. Other appointments 
include chair of the Victorian Qualifications Authority 2004 to 2007 and later a 
member of the Victorian Registration and Qualifications Authority. He was 
executive director of the Monash University-ACER Centre for the Economics of 
Education and Training (CEET) until 2008 and now has an adjunct appointment at 
Monash as a professor in education. His research interests are in financing, 
participation and quality in education and training, and education and 
employment. 

■ Professor David Finegold, Chief Executive Officer, American Honors Organisation 

David Finegold is currently Chief Academic Officer for the recently established 
American Honors Organisation; taking two years leave from his role as Senior VP 
for Lifelong Learning and Strategic Growth at Rutgers, the State University of 
New Jersey. At Rutgers he was responsible for continuing and executive 
education, online learning, international and off-campus programs, and 
strengthening relationships with companies and other stakeholders. Dr. Finegold 
is a leading expert on skill development systems and their relationship to 
economic performance. His current work is focused on the evolving skill 
development systems in India and China. Since arriving at Rutgers in 2006, Dean 
Finegold has spearheaded efforts to build a workforce development system for 
New Jersey’s bioscience sector. Prior to joining Rutgers he was a professor at the 
Keck Graduate Institute of Applied Life Sciences in Claremont, CA, where he 
helped to build the first college devoted specifically to creating a best-in-class 
professional science masters program. He is the author of more than 80 journal 
articles and book chapters and has written or edited seven books, including 
Transforming the US Workforce Development System and Are Skills the Answer? 
He graduated summa cum laude with a BA in Social Studies from Harvard 
University in 1985, and was a Rhodes Scholar at Oxford University, where he 
completed his DPhil in Politics in 1992. He was elected into the New Jersey High 
Tech Hall of Fame in 2008 for the leadership he provided in launching Bio-IStop, 
which has evolved into the talent network for the state’s bioscience industry, 
and creating the Master of Business and Science, the first common inter- 
disciplinary degree framework spanning schools on all three Rutgers campuses. 

■ Dr Robin Ryan, Adjunct Lecturer, Graduate Program in Educational Leadership and 
Management, School of Education, Flinders University 

Robin Ryan is a consultant in vocational education policy and research and 
adjunct lecturer at Flinders University, Adelaide. A former Assistant Director for 
Policy in SA TAFE, where he was centrally involved in the national vocational 
education reform agenda, Robin is also active in the Australian College of 
Educators, especially as a member of the Archives Research Fellows Group. 
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Professor Kwong Lee Dow, AO, Professor, University of Melbourne 

Kwong Le Dow has worked at the University of Melbourne since 1966. His roles 
included Dean of Education, Deputy Vice-Chancellor and Vice-Chancellor. 
Additional appointments include Chair of the Victorian Curriculum and 
Assessment Authority, member of the national Higher Education Council, Chair of 
a national Review of Teaching and Teacher Education, Chair of a Review of 
Student Income Support Reforms (2011), and Chair of a Review of Higher 
Education Regulation (2013). Other work for the Victorian Government has 
focused on the connections between universities and TAFE institutes (2009-11), 
particularly in regional communities. 

Dr Tom Karmel, Adjunct Professor, National Institute of Labour Studies, Flinders 

University 

Tom Karmel is a former Managing Director of the National Centre for Vocational 
Education Research, retiring in 2013 after being with the company since August 
2002. Prior to this position he held senior appointments in the federal 
government areas of education, employment, labour market research and the 
Bureau of Statistics. His research interests have centred on the labour market 
and the economics of education, focusing on empirical modelling, and he has a 
particular interest in performance indicators, both in higher education and 
vocational education and training. He has an honours degree in mathematical 
statistics (Flinders), and a Masters of Economics and doctorate from the 
Australian National University (The impact of increasing education levels on the 
Australian workforce). Currently, Dr Karmel is President of the Economics Society 
of Australia, South Australian branch, and an adjunct professor at the National 
Institute of Labour Studies, Flinders University. 
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